Caching a model’s context is not a free speed toggle. It is a decision about what staleness you are willing to tolerate, and treating it as an afterthought is how a session starts answering from a world that no longer exists.
When an API offers to cache part of a prompt so a repeated call does not have to reprocess it from scratch, it reads like a pure win: same output, less work, so why would you not turn it on everywhere it is offered. I turned it on everywhere it was offered for a while, on exactly that reasoning, before running into the case that changed my mind. A long-running agent session had a file’s contents sitting in the cached prefix, the file changed under it, and the model kept answering questions about the old version for longer than it should have, because the cache had no way of knowing the world it described had moved on.
The mechanism is not a bug in any particular tool, it is what caching means by definition. A cache is a bet that the thing being reused has not changed since it was captured, and every caching decision is really a decision about how much staleness you are willing to tolerate in exchange for not redoing the work. That trade-off is fine, often good, when the underlying thing is genuinely stable for the duration you are relying on it: a system prompt, a set of instructions, a reference document that is not being edited mid-session. It stops being fine the moment the cached thing is something that changes during the exact window you are trusting the cache to cover, and nothing about the caching mechanism itself is going to notice that for you.
What makes this easy to miss is that the failure is quiet rather than loud. A stale cache does not throw an error, because from the system’s point of view nothing went wrong: it did exactly what caching is supposed to do, serve the previous result instead of recomputing it. The model answers fluently, using the old file, the old state, the old assumption, and there is no signal anywhere in that fluent answer telling you it is reasoning about a world that no longer exists. I only caught mine because the answer referenced a function I had already deleted, which was specific enough to be obviously wrong. A subtler version of the same staleness would have gone through unnoticed.
The fix is not to stop caching, which throws away a genuine and often large efficiency gain for no reason. The fix is treating the cache boundary as a real design decision with an owner, rather than a setting you turn on because it is available: deciding explicitly what is allowed to live behind the cache because it is genuinely stable for the relevant window, and what has to be excluded, or re-fetched fresh, because it can change under you during that same window. A system prompt earns a long-lived cache easily. The live contents of a file an agent might itself be editing during the session does not, and putting it behind the same cache as the stable material is where the actual mistake happens, not in caching itself, but in caching the wrong boundary.
I now ask the same question every time a caching option is offered to me, at any layer of a system: what has to stay true for the cached duration for this to still be correct, and am I actually confident that condition holds. Most of the time the answer is yes and the cache is free efficiency exactly the way it was advertised. Occasionally the honest answer is that the thing behind the cache can move during the window I am relying on it, and in that case the right call is to pay the recomputation cost rather than serve an answer that is fluent, confident, and quietly describing a version of the world that is not there any more.