Tests pass in a commit message is a claim. A committed verdict file ending VERDICT: PASS is evidence, and the difference changed what done means in my pipeline.
For a long time I accepted a commit message that said ‘tests pass’ as though it were the same thing as tests actually passing. It rarely was a deliberate lie. An agent runs a suite, sees green, writes the summary, and moves on. But the summary is a claim made after the fact, disconnected from the run that produced it, and nothing stops the next session, or the next agent, or me a week later, from trusting a claim that was true once and is not true any more.
The fix I landed on is unglamorous: proof has to be a file, not a sentence. Every task that finishes in my pipeline produces a committed artefact under a proof directory, and the artefact is not a description of a result, it is the result. A gate run writes its verdict, ending in a line that says VERDICT: PASS or VERDICT: FAIL, and that line is what a downstream check parses. Nobody reads the prose above it and decides whether to believe it. The parser reads the terminal line or it does not find one, and a missing or malformed verdict is treated exactly like a fail.
This changes what done means in a very specific way. A task is not done because an agent says it finished. It is done when the proof file exists on disk, matches the commit it claims to certify, and carries a verdict from a run that actually happened. If any of those three things is missing, no file, a file for the wrong commit, or a verdict that reads as pending, the task is not done, whatever the chat transcript says.
The commit-matching part turned out to matter more than I expected. A proof file written against an earlier version of the diff is not proof of the current one. I had a task pass its gate, then pick up one more small edit before it shipped, and the proof file from the earlier round was still sitting there looking convincing. Pinning the proof to the tree hash it was written against, and re-checking that hash before trusting the file, closed that gap. Without the pin, a stale pass is indistinguishable from a current one until something breaks.
The other benefit is that proof survives the session that produced it. A claim made in a transcript is gone the moment the context window rolls over or the session ends; nobody two weeks later can go back and re-read what an agent asserted in a conversation nobody kept. A file in the repo is there for as long as the repo is there. Anyone auditing the estate later, human or agent, can open the same file and see the same evidence, rather than being asked to trust a summary of a summary.
None of this is a novel idea, it is just applying the same standard to an AI-generated claim that any serious engineering organisation already applies to a human one. Nobody accepts ‘I tested it’ from a person without a CI badge or a test report to back it up. The only reason it took me a while to enforce the equivalent for agents is that their prose is fluent enough to read as evidence even when it is not, and fluency is exactly the property that should make you more suspicious of an unverified claim, not less.
What I check for now, before I let anything ship, is boring and mechanical: does the file exist, does it match the current commit, does it end in the word PASS. If any answer is no, the task goes back, regardless of how confident the write-up sounds.