5xx errors, 429 and maintenance: how crawlers react

A site can have excellent content and an impeccable structure, but if the server responds with errors when the crawler comes by, the engine sees none of it. Server-side errors are a signal engines take seriously: they show that the site can't cope with the requests, and their natural reaction is to slow down. If the errors last, pages can drop out of the index.

The codes to know

All 5xx codes say the same basic thing: the problem lies with the server, not the page. 503 and 429 add a useful piece of information: the situation is temporary.

How engines react

Faced with frequent errors, crawlers:

1. reduce the crawl rate so as not to overload the server;
2. retry later the URLs that responded with an error;
3. if the error persists for a long time, may remove the pages from the index, because they no longer have confirmation that they exist.

The delicate point is that, after the fix, the crawl rate rises again only gradually. A period of errors can therefore slow down index updates for some time even after the problem has been solved.

The special case of robots.txt

Before crawling a site, the crawler requests robots.txt. If that file returns a 404, the engine treats the site as unrestricted. If it returns a 5xx error, however, the engine doesn't know which rules apply and, as a precaution, may suspend crawling of the entire site until the file becomes available again. A robots.txt served by a slow or fragile application is therefore a weak point: it's better to serve it as a static file.

Scheduled maintenance

When a site has to go offline for an update, the correct response is:

HTTP/1.1 503 Service Unavailable
Retry-After: 3600

• 503 communicates that the unavailability is temporary;
• Retry-After says after how many seconds, or at what date and time, to retry.

Mistakes to avoid:

• showing a "site under maintenance" page with a 200 code: the engine crawls it as normal content and may replace, in the index, the real content of every page with it;
• redirecting all URLs to a maintenance page with a 302;
• keeping the 503 up for days: it's meant for short interruptions. Prolonged unavailability is treated as a persistent problem.

In my work I prefer to avoid switched-off maintenance altogether: with a well-organised release, preparing the new version alongside the old one and then swapping them, downtime shrinks to a few seconds or to zero.

The 429 code and rate limiting

A rate-limiting system protects the server from abuse, but it must be tuned so as not to hit legitimate crawlers. Responding 429 to a crawler asks it to slow down: that's fine as a temporary measure, but if it becomes the norm, crawling is reduced permanently.

If the server struggles to keep up with crawlers, structural solutions are better:

• caching the most requested pages;
• reducing pointless URLs to crawl, such as parameters and filters;
• static resources served from a CDN;
• 304 Not Modified responses to conditional requests.

When the firewall is doing the blocking

A frequent problem that's hard to see: the server works perfectly, but an application firewall, an anti-bot protection or a CDN responds to crawlers with a 403, a challenge page or an error. From the administrator's browser everything looks fine.

Typical signs:

• webmaster tools show crawl errors that nobody can reproduce;
• in the CDN logs, crawler requests receive different responses from those of users;
• the URL inspection tool shows a challenge page instead of the content.

The solution is to explicitly allow verified crawlers, identified through reverse DNS or the address lists published by the engines, and not just by user agent.

Monitoring

• Crawl stats report in webmaster tools: shows responses by code and average times.
• Server and CDN logs: filter crawler requests and count errors per day.
• External uptime monitoring, with immediate alerts in case of errors.
• Response time: a server that slows down often comes before a server that fails.

In short

• 5xx errors slow crawling down and, if they persist, push pages out of the index.
• A robots.txt that returns 5xx can block crawling of the whole site.
• For maintenance use 503 with Retry-After, never a page with a 200 code.
• 429 is a temporary brake; heavy loads call for caching and fewer pointless URLs.
• Make sure firewalls and CDNs don't block legitimate crawlers.