I overrode a worker’s grep of the real data with a better-sounding principle once, and shipped the regression it had already warned me about.
A worker flagged a change I’d asked for, twice, with the same piece of evidence both times: it had actually looked at the data the change would touch, and found one entry that needed the old behaviour preserved and none that needed the new one. My response was a principle — the new behaviour was more correct in general, the one exception looked like an edge case, and edge cases get handled, not used to block a rule that’s right everywhere else. It was a reasonable-sounding argument. It was also wrong, and I only found out because the entry it had flagged was a real, already-committed piece of data that broke the moment the change shipped.
What strikes me now isn’t that I made a mistake — that happens constantly and isn’t interesting on its own. It’s the specific shape of the mistake: I had a measurement of the actual corpus, produced by something that had gone and looked, sitting next to my own reasoning about what the corpus was probably like in general. And I let the reasoning win, because it sounded more like expertise. A principle explains why something should be true. A grep tells you whether it is true, in the one place that actually matters, which is the data you’re about to ship a change against.
This is a bias with a very specific trigger: it fires hardest exactly when the principle is genuinely good. If the reasoning is obviously weak, nobody overrides a measurement with it — the conflict is easy to resolve. The dangerous case is a well-reasoned, generally-correct argument meeting one small, concrete, contrary fact, because the argument has the rhetorical weight of sounding like the more sophisticated position. Overriding evidence with a story about why the evidence probably doesn’t generalise is something that feels like judgement in the moment and reads as recklessness afterwards, and the only way to tell which one it is, is to check.
The check I’ve settled on is asymmetric on purpose. To overrule a measurement, I need a different measurement — a wider grep, a re-run against more data, an actual look that turns up something the first look missed. “The principle is right” doesn’t clear that bar, no matter how right the principle is, because the principle was never in dispute. What was in dispute was whether this specific case was the exception the measurement said it was, and a rationale about the general case doesn’t answer a question about a specific one.
I’ve since watched the same pattern from the other side — a reviewer citing a best-practice rule against a change that a direct test of the running system had already validated, and losing the argument correctly, because the test outranked the rule. That’s the version of this that should happen more often: not blind deference to whoever ran a command, but a standing rule that a real measurement of the actual thing beats an argument about what the thing is probably like, including when the argument is mine, and including when it’s good.
It applies past code, too. A reviewer who says “this is fine, I checked” has made a claim of the same shape as the worker’s grep, and it deserves the same scepticism until it comes with what was actually run and what it actually showed — the phrase “I checked” is not itself a measurement, it’s a promise that one happened, and promises are exactly the thing this whole discipline exists to stop accepting at face value.
The regression I shipped got fixed within a day, so the cost was contained. What stayed with me was smaller and more durable — the moment where a fact I had access to lost to a sentence I found more persuasive, and how easy that trade was to make without noticing I’d made it.