TL;DR: One of the best tips to improve quality and reduce token
consumption when writing software with AI is to keep your AGENTS.md (or its
equivalent) up to date.
I have flown back into Malta, after a week of training multiple software teams in Georgia (the country, not the state) about generative AI and I am, as usual, swamped with the amazing amount of advances and research coming in from all directions. I typically share these thoughts on LinkedIn, but my friend Andrew Sundstrom basically said, “knock it off” (opens in new tab). I should be writing stuff here, where it’s easier to find.
Yeah, OK. I’ll go with that. So now I have a new tag, trench, for posting these shorter “life in the trenches” things. I suspect this will contain useful information about genAI, and mostly from the standpoint of software engineering. So I crawled out of bed this morning and started writing.
But ...
You’re swamped with “AI-influencer” slop:
- “AI won’t replace you. A person using AI will.”
- “We’re not in an AI bubble. We’re in an AI revolution.”
- “Prompting is the new coding. Learn it or get left behind.”
Any of those statements may or may not be true, but none are actionable. Frankly, I have too much damned self-esteem to humiliate myself like that, singing and dancing publicly in this AI vaudeville, hoping to make the big time. Thus, the “trench” posts will be about reality, not some emoji-laden post for my non-existent newsletter. So my rule of thumb for “trenches” articles is clear: if the article isn’t actionable, it doesn’t get written.
Steering Files
When you’re using some agentic coding tool, such as Claude Code (my favorite),
Kiro, Antigravity, or something else, you have what this arXiv preprint refers
to as “instruction files” (opens in new tab): CLAUDE.md,
AGENTS.md, and so on. When you start coding, the AI reads these files (which I
call “steering”) to ground itself in your codebase. The files should generally
contain four things:
- What your software does
- What technologies it uses
- The directory structure (how to find behavior)
- Hard rules that the AI will never violate (stop laughing)
But there’s a huge problem with them and I keep hitting this with software teams: they’re not keeping these files up to date.
These files tell you where you are, not where you’re going to be in three months or six months. This means that three or six months later, these files are out of date if they’re ignored, and I’m finding that they are almost always ignored. There is a simple fix, but I got it wrong at first.
But stepping back a bit, why is this so important? I was working with one team
that complained bitterly about the quality of the code AI was producing. In
fact, the code often wouldn’t even compile. They were not happy using genAI.
So I arranged a pair programming session. One of the first things I had them do
was to run PAAD’s (opens in new tab) /pushback skill on their
steering files.
The /pushback skill basically takes any document and looks for flaws,
omissions, bad ideas, ambiguities, and so on. So they ran this skill and one of
the first things they found was that their steering files had a critical
portion of their tech stack wrong. They claimed they were using a modern,
popular
Web framework rather than the older, less well-known framework they were using.
So the AI was grounding itself in a lie and was writing the wrong code.
This and a few other fixes completely turned things around. Now they’re quite happy with AI. In fact, at one point, due to a bug in their IDE, they were locked out of using Opus models, but later reported to me that using Sonnet with PAAD was better than using Opus without PAAD. I call that a win.
But what did I get wrong? In the early days of my training, I was explaining to the teams that they needed to regularly review the steering files to ensure they were up to date. You don’t want cowboys pushing changes to steering all the time and possibly messing things up. That sounded reasonable to me. I was an idiot (still am, at times).
The problem was that I was basically asking them to schedule regular meetings to handle an incredibly important yet boring task. Not gonna happen.
But remember that arXiv paper I referenced earlier? They found not only that AI tended to assume that steering files were true, but also that it would often rewrite existing, working code to match the steering files, even if those files were wrong. Their fix was to add one sentence to the steering files:
When a repository’s documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code.
From the paper:
We added one sentence to the user’s own instruction file, stating that existing code takes precedence when documentation and code disagree. Across 40 projects and two agents, this moved the agents from following the stale file in most choices to following the code in nine out of ten, and no agent rewrote existing code.
Note the “stale file” bit. That’s the real problem. While I generally like their fix, I’ve found for personal projects that there’s an even better fix. Include the following in steering (adjust to match your project needs):
Before pushing a pull request, reread CLAUDE.md and determine if it should be updated to reflect changes in the code. Discuss proposed changes with the developer. They must approve all changes.
Your steering files are living documents and need to be updated regularly. I’ve found that this one addition allows me to keep my steering files accurate and improves the overall quality of the code. The “one-sentence” fix the authors list is great, but it doesn’t explicitly guide the developers towards updating the steering files. In working with many teams, I’m still finding that they often don’t read the full output from the AI, so an interactive “let’s fix this” is more likely to be followed than the AI simply dumping a wall o' text.
So keeping your steering files up to date not only gives you better-quality code, it also reduces your token usage. Instead of hunting for answers to common questions—and sometimes getting the answers wrong—you hand the AI the answers on a silver platter.
When you’re getting software teams up to speed on using agentic tools to write software, fixing your steering files must be your first step.