A model’s visible reasoning trace is generated the same way its final answer is, one token at a time, which means it can look like a proof of the working and still not be one.
When a model shows its working before giving an answer, it is tempting to read that trace the way you would read a person’s working on a sum: a faithful record of the steps that actually produced the result, laid out so you can check each one. I used to read it that way myself, right up until I started catching cases where the visible reasoning and the actual answer did not line up: a trace that walked through one path, arrived at a conclusion, and a final answer that quietly took a different one, with nothing in the output flagging the mismatch. The gap was subtle enough each time that it took several repeats before I trusted the pattern rather than assuming I had simply misread one particular example.
The mechanism explains why this happens without needing anything mysterious. The reasoning trace is not a log retrieved from some internal process that separately computed the answer. It is generated the same way the final answer is, one token at a time, each token conditioned on everything before it, including its own earlier reasoning tokens. That means the trace can shape the eventual answer by being there, since the model conditions on what it just wrote, but it is not guaranteed to be an accurate report of some other, hidden computation that really produced the answer. There may be no such separate computation to report on in the first place. The trace and the answer are, in that sense, two outputs of the same single process, not a working shown and then a result separately checked against it.
The nearest human comparison I have found useful is confabulation rather than deliberate deception. A person asked to explain a snap judgement will often produce a fluent, plausible-sounding account of their reasoning that feels true to them and is not, in fact, an accurate description of what actually drove the decision. The account is generated after, or alongside, the judgement, in the same fluent language-producing part of the mind that does everything else, and it is honestly offered. It is just not necessarily what happened. A model’s reasoning trace can be exactly that: an honestly generated, fluent, plausible account that happens not to be the actual causal story behind the answer sitting next to it. I have caught myself, reading a particularly fluent trace, half persuaded by its confidence before I had actually checked a single step in it.
This matters practically because it changes what a visible reasoning trace is good for. It is not proof that the answer is correct, and it is not a reliable log of the actual computation. What it still is, usefully, is a hypothesis about the answer that I can go and check independently: read the steps, see if they hold up on their own terms, verify the specific claims made along the way, rather than accepting the trace’s existence as evidence that the answer underneath it is sound. A long, careful-looking chain of reasoning attached to a wrong answer is not a contradiction that needs explaining. It is the expected failure once you stop assuming the trace and the answer share one faithful causal path. Reading the trace and verifying the conclusion are two separate jobs, and only doing the first one is how a confident wrong answer gets past me.
None of this makes the reasoning trace worthless, and I still find it useful most of the time, because a trace that is wrong somewhere is usually wrong in a way I can spot once I actually read it rather than skim it for reassurance. The mistake is treating its presence as a substitute for that reading, assuming that because a model showed its working, the working must be why the answer is right. Showing the working and being able to prove the working caused the answer are different claims, and only the first one is actually guaranteed by how these models generate text.