A two-point gap on a public coding benchmark tells you almost nothing about your own repository. The only eval that predicts your outcomes is one built from your own tasks.
Every few weeks a new benchmark leaderboard makes the rounds, and every time, a small delta between two models gets treated as though it settles which one to actually use. It does not settle anything, because the benchmark was not built on your repository, your dependency versions, your test conventions, or the specific way your codebase tends to break. A model that scores marginally higher on a general coding benchmark can still be measurably worse at the actual work you hand it, and the leaderboard has no way to tell you that, because it was never measuring your work in the first place.
The mismatch comes from what benchmarks have to optimise for to exist at all: broad applicability across many labs, many tasks, many languages, so that the number means roughly the same thing to everyone looking at it. That breadth is exactly what makes the number weak evidence for a specific decision. Your stack is narrow. Your failure modes are narrow and specific to how your team writes code, which frameworks you lean on, and which categories of task you actually hand an agent day to day. A benchmark averaged across everyone is, by construction, not describing anyone’s specific situation particularly well.
I have watched this play out directly: a model that led a popular leaderboard performed worse on my own tasks than a model ranked below it, because the leaderboard’s task mix leaned towards categories of problem I rarely hand to an agent, while under-representing the category I hand it constantly — multi-file refactors inside an existing pattern, where following convention correctly matters more than raw problem-solving cleverness. Switching back to the higher-ranked model on the strength of the leaderboard alone would have made my actual output worse while every public number said I was upgrading. The leaderboard was not wrong about what it measured. It was just measuring something other than what I needed to know.
What actually predicts an outcome for me is an eval built from my own tasks — real pieces of work I have already done, with a known-good answer or a known-good outcome, run against whichever model I am deciding between. It is more effort to build than reading a public number, and that effort is exactly the point: a general benchmark is cheap because it answers a general question, and the question I actually have is never general. It is whether this model will do well on the kind of thing I hand it, and no leaderboard was constructed to answer that.
This is not an argument against benchmarks existing, and it is not a claim that they are meaningless — they are a reasonable first filter for which few models to bother evaluating properly, and a useful signal when the gap between two models is large rather than marginal. The mistake is treating a small, general delta as a specific, operational answer, the way a single averaged number from a population of strangers gets treated as a forecast of what will happen to you personally. A narrow gap on a broad benchmark tells you almost nothing about which model will save you time on your actual backlog, and that gap gets treated as decisive far more often than the evidence behind it justifies.
The honest version of "which model should I use" is never answered by someone else’s leaderboard, because someone else’s leaderboard was never trying to answer it. It is answered by running your own tasks through the candidates and reading what actually comes back, and until you have done that, the leaderboard number you are quoting is a proxy for a question you have not actually asked yet.