RFC 9309 treats a robots.txt fetch that fails with a 4xx and one that fails with a 5xx as opposite cases: a 4xx means the crawler may proceed, a 5xx means it must assume complete disallow, and a crawler that merges every non-200 into one branch gets that backwards.
My crawler had one branch for whenever robots.txt did not come back as a normal 200, and it treated every non-200 response the same way: log it, assume no restrictions apply, carry on crawling. That branch survived unnoticed for a long time because most of the failures it caught really were the harmless kind, since a site with no robots.txt at all returns a 404 and there is nothing there to obey. It stopped being harmless the day a fetch hit a site that was mid-outage, returning 503s, and my crawler carried on exactly as if nothing had been asked of it.
RFC 9309, the Robots Exclusion Protocol, does not treat those two failures the same, and reading it properly is what caught my own mistake. On a 4xx response, the crawler may access any resources on the server, because a 4xx means the file genuinely is not there and there is no stated restriction to violate. On a 5xx response, the rule flips, in stronger language: the crawler must assume complete disallow, as if the file existed and forbade everything, until the server recovers enough to actually say otherwise.
The reason those two get opposite treatment, once I actually thought it through instead of merging them into one could not fetch it case, is that they mean different things about the site’s intent. A 404 is the server reliably telling you there is no robots.txt, which is an answer, just not a restrictive one. A 5xx is the server failing to answer at all, and a crawler has no way to know whether the file that would have loaded, had the server been healthy, was empty or full of disallow rules for exactly the paths it is about to hit.
My branch had no idea any of this distinction existed. It checked for 200 and treated everything else, a 404, a 500, a 503, as one undifferentiated no file, proceed case, which is the may-access behaviour applied unconditionally to a situation where the must-disallow behaviour was the one actually required. For as long as an outage like that lasts, a crawler written the way mine was is outside what the specification permits on every request it makes, and nothing in its own logs makes that visible, because from its point of view every one of those requests looks like an ordinary missing file.
The fix was to stop collapsing status codes into a single non-200 bucket and branch on the actual range: a 4xx clears the crawler to proceed, a 5xx stops it cold until a later fetch either succeeds or comes back with a genuine 4xx. It is a small amount of code, and it only mattered because of how confidently wrong the old branch had been, not failing loudly, just quietly doing the opposite of what the specification requires exactly when that requirement mattered most. Nothing about the fixed version would have shown up in an ordinary test either, because a normal test suite does not simulate a dependency returning 503s at exactly the moment a crawl is underway, which is precisely why the bug lived as long as it did.
The same collapsing habit exists in more of my code than just that one crawler, once I started actually looking for it: anywhere I had written a single catch for anything other than success, without asking whether the different failure modes underneath it actually call for different responses. A 4xx and a 5xx are not two flavours of the same problem. One tells you something definite about the resource, the other tells you the server could not tell you anything at all, and treating an absence of an answer as equivalent to a confirmed one is the same mistake wearing a different status code range each time it turns up.