To keep an AI coding agent from repeating the same mistake, instead of editing CLAUDE.md by hand every time it slips, you can automatically collect “what the user corrected” from the session logs and use it in two ways. One is to show the recent corrections as they are at the first input of the next session in that repository (learning from the very first correction); the other is to raise a rule candidate to the proposal ledger when the same type appears on two or more different days in the same repository (hardening it into a sentence). Claude Code leaves sessions as JSON Lines (.jsonl), so you only need to filter user messages with regular expressions for shapes like “you missed it” or “I told you not to,” and attach a type.
This post is the fifth in a series that covers, one per day, the devices I built while turning my solo development environment around AI agents (AX, AI transformation). The last one was the story of a hook that stops commits with no review record.
Why was “edit CLAUDE.md when it slips” not enough?
The advice to tell the agent “edit the rules file so this mistake is not repeated” whenever it slips is widely known, and I started that way too. Two things got in the way.
First, the one doing the editing is a person. Corrections pop out briefly in the middle of work. After saying “no, that’s not it” or “you left it out,” I go back to work, and opening the rules file to polish a sentence gets postponed. With several repositories, I even have to wonder which file to write it in.
Second, a correction made once disappears when that session ends. The model remembers nothing after the conversation ends. When I opened a new session in the same repository the next day, it made the same mistake on the same assumption, and I said the same thing again. The gap before something becomes a rule was the problem.
Where do the corrections remain?
Claude Code leaves every message and tool call of each session in ~/.claude/projects/<working folder>/<session id>.jsonl. The user messages are in there as they are. So if you scan these files over a date range, pull out only the user messages, and pick only those shaped like “a correcting remark,” you have correction cases. The agent’s work just before the correction (which tool it used to do what) is in the same file, so you can also keep a gist of “what was wrong.”
At first there was a metric that measured only the “correction rate.” The rate tells you whether things got worse but not what went wrong, and it did not give grounds for editing the guidelines. So on 2026-10-10 I built a separate script that collects the sentences themselves as cases.
Which remarks count as a “correction”?
The part that took longest here was the filtering criteria. Remarks such as “check again” or “isn’t it ~?” are often not about a mistake. When I looked at a 30-day sample by hand, half were this kind of re-check or doubt. If even those get raised as rules, wrong rules pile up and ruin the agent further. So I put precision first and narrowed the types accepted by rule to four.
TYPES = ( # first match from the top. If it ends with a question mark, only "violated instruction" is accepted
("누락", re.compile(r"놓쳤|빠졌|빠뜨|빼먹|누락"), False),
("지시 위반", re.compile(r"말라(?:고|니까|니깐|했)|말랬|하지\s?말(?:라|랬|래|자)|하라고\s*[.!]*$"), True),
("전제 착오", re.compile(r"(?:그게|그거|이게)\s?아니|[가-힣A-Za-z0-9_.]+(?:이|가|게)\s?아니(?:고|라)(?=\s)"), False),
("결과 오류", re.compile(r"잘못(?:됐|되었|했|만들|계산|나왔)|틀렸|틀린|이상해|엉뚱|엉망"), False),
)Remarks that do not match any rule are only collected as “other.” Exceptions were added on top of that. “I said it wrong” is the person’s own slip, so it is excluded; “if it goes wrong” is a hypothetical, so it is excluded; and a remark that explains after agreeing, like “yes, that’s not it …,” is excluded too. Messages sent by another session (starting with ”[”) and long instructions over 200 characters are not corrections either. “Other” is later sorted again into one of the four types by asking a small model once more, but the default is “not a correction.” One wrong correction does more harm than ten right ones.
Where do I use the collected cases?
There are two branches.
First, I show them at the first input of the next session. A hook (UserPromptSubmit) that runs when a session’s first prompt arrives picks, from that repository’s corrections of the last 30 days, only those with a decided type, up to 4, adds 2 mistakes that a review agent or a guard hook caught on its own, and injects them once. A marker file keyed by session id prevents putting them in the same session twice. This way, even before it becomes a rule, the next session starts out knowing from the very first correction.
[Recent corrections] These are remarks the user corrected in this repository
within the last 30 days, and mistakes found by review or by the agent itself.
Before making the same assumption or the same mistake, first check against this list —
when citing one as evidence, check the original text again:
- 2026-10-20 missed item: …
- 2026-10-18 violated instruction: …Second, repeats are raised as rule candidates. If cases of the same repository and type occur at least twice in “units of different sessions and dates,” they get a number and are raised to the proposal ledger. A repeat in the same session on the same day counts as one, because it may be the same problem said several times. A bundle where no human correction exists and only things the agent found by itself have piled up is not raised but put on hold. The more a session calls reviews often, the more proposals poured out without a hold. The ledger holds only repository, type, session count and dates, and the original text stays local.
# One line the nightly automation runs — collect the last 4 days of cases, re-sort "other" with a model, then raise repeated types to the ledger
python3 scripts/corrections-collect.py --days 4 --llm --propose
# View the cases of a raised bundle (local screen only)
python3 scripts/corrections-collect.py --show <repo>/<type>How has my day changed?
These are values measured as of 2026-10-23. The cases collected from 2026-09-11 through 2026-10-21 number 363: 135 are remarks corrected by a person, and 228 are mistakes that a review agent, a guard hook or the agent itself noticed first. By type, violated instruction is 80, wrong result 63, missed item 48 and mistaken premise 46, and what remains as “other” is 126. The first-input injection went in 297 times across 141 sessions since 2026-10-11. In the proposal ledger, 32 “repeated correction” items were raised, and 27 of them were reflected as rule sentences or hooks.
Two things changed. I no longer worry about where to write a correction. If I say a word in the middle of work and move on, that evening’s automation collects it, and the next day the session in that repository starts out having seen that remark. And the sentences that go into the rules file are limited to “things repeated at least twice,” so rules grow less often because of something that annoyed me once. There is a cost too. Because it is regex-based, “other” is the largest bundle, and real corrections hidden in it have to wait until the model re-classification catches them.
Frequently asked questions
Where are the session log files, and what do they contain?
Claude Code puts a .jsonl file per session under ~/.claude/projects/<encoded folder name>/ for each working folder. One line is one message, and user messages, agent messages, tool calls and their results are in order. The script in this post reads only the user messages and the tool call just before.
Can I push the original correction text to git?
I do not. Correcting remarks mix in data being worked on, file names and sometimes secrets. The case files stay only in a local folder, and for sentences that contain secrets or SQL, only a masked marker remains instead of the original. The proposal ledger that goes to git holds only repository, type, session count and dates.
Why not let a model classify everything from the start?
Because of precision. If a wrong correction becomes a rule, the agent tries to obey a wrong rule. So I accept only the four rule-based types first, and use the model only to re-sort “other.” Even then, one model’s judgment counts with the same weight as a rule-based judgment only when another model sorting the same question again agrees with it. If they disagree, it goes back to “other.”