Token count is one line item, not the total. The honest number is everything a change cost from first attempt to merge — retries, review rounds, and the human minutes spent unblocking it.
Ask most agent dashboards what a feature actually cost and you get a token count, if you get anything at all. Token count is not cost, it is one line item inside cost, and treating it as the whole answer hides the parts that usually dominate: the retries when the first attempt failed a test, the review rounds where a human or a second agent sent the change back, and the human minutes spent reading, correcting, and re-prompting until the thing was actually mergeable rather than merely produced.
I think the honest metric is cost per merged pull request, counted end to end: every token spent across every attempt that led to that specific merge, including the attempts that failed and got thrown away, plus review time, plus the wall-clock time a human spent unblocking it. A change that took one clean pass and cost a small number of tokens is genuinely cheap. A change that took six retries, two full review rounds, and forty minutes of a person’s attention to finally get across the line is not cheap just because the token bill for any single one of those six attempts looks small in isolation — the real cost is the sum, and most tooling reports the smallest visible piece of that sum as though it were the total.
Even when a token price falls, I do not treat that as evidence that the whole activity has become cheaper; those are different claims. If a task now takes twice as many retries to reach a mergeable state as it did before, because the work being attempted has become more ambitious, cheaper tokens can still produce a more expensive merged PR than before, once you count the full path rather than the per-token rate. A dashboard that stops at the token meter cannot answer that question. I want mine to track the full path before I use its numbers to decide whether the setup is paying for itself.
Retries are the part I have found most consistently invisible. An agent that fails a test, reads the failure, fixes it, and passes on the second attempt looks, from a token-spend view, like a single successful task that happened to use somewhat more tokens than average. From a cost-per-merged-PR view, it is a task that cost two attempts, and if that pattern repeats across most of a week’s work, counting attempts as completed work overstates delivery. Two attempts for one merged change mean one result, but not necessarily twice the time or cost: a small retry and a full rebuild have to be measured separately.
Review rounds are the part that is hardest to attribute cleanly, and also the part I refuse to leave out, because it is real time and real cost even when nobody logs it against the task. A change that clears an independent review on the first pass and one that bounces back twice before it clears cost visibly different amounts, in the reviewer’s time and in the additional build cycles the second one needed, and a metric that ignores this difference is measuring how much was produced, not how much a finished, mergeable, reviewed change actually cost to get there.
None of this is an argument against using agents for real work — quite the opposite, it is the argument for measuring the thing that actually determines whether the economics work. Cheap tokens and an expensive true cost per merged PR can coexist, and most reporting is built in a way that would never let you notice. The number worth trusting is the one that adds up everything it took to get a change from opened to merged, in pounds and in the hours a person actually spent on it, not the number that was easiest to expose on a dashboard.