The term crawl budget gets used a lot, often inappropriately. It is invoked to explain any indexing problem, even on sites with a few hundred pages where it plays no role at all. In reality it is a precise concept, useful in precise contexts, and knowing when to apply it avoids pointless work.
What it is, in practice
Crawl budget is the number of URLs a search engine is willing and able to crawl on a site in a given period. It comes from the combination of two factors.
- Crawl capacity: how many requests the crawler can make without straining the server. If the host responds quickly and without errors, the limit rises; if it slows down or returns 5xx errors, the crawler backs off.
- Crawl demand: how much the engine wants to visit that site. It depends on how popular the URLs are and how often the content changes.
The result is a number of visits you cannot buy or configure, but that you can spend well or badly.
When it is worth worrying about
For a site with a few hundred or a few thousand pages, whose content rarely changes, the crawler can normally visit everything without difficulty. If a page is not indexed there, the cause almost always lies elsewhere: quality, duplication, noindex, lack of links.
Crawl budget becomes relevant in three situations:
- very large sites, such as e-commerce catalogues, classified ad portals, editorial archives;
- sites whose content changes often, where freshness matters and crawl delays translate into outdated results;
- sites that generate near-infinite URLs, for example with combinable filters, calendars or session parameters.
In the third case even a site that is «small» in terms of products can produce millions of distinct addresses, and there the problem is real.
Where the budget gets wasted
The most common forms of waste are technical and repetitive:
- faceted navigation that creates a URL for every combination of filters (colour, size, price, sort order);
- tracking or session parameters added to internal links;
- redirect chains, where each hop is one more request;
- error pages that return 200, which the crawler keeps visiting as if they were valid;
- duplicates caused by variants with and without
www, with and without a trailing slash, in upper and lower case; - infinite spaces, such as calendars with a «next month» link that never ends.
Every request spent on these addresses is a request not spent on the pages that matter.
How to intervene
Reduce useless URLs at the source
The best solution is not to generate the superfluous URL in the first place. Filters that do not deserve an indexable page can be handled without changing the address, or with links the crawler does not follow. Tracking parameters should be removed from internal links.
Block what must not be crawled
For sections that should never be visited, such as internal search results or worthless filter combinations, a robots.txt rule prevents crawling. Be careful, though: a blocked URL cannot communicate a noindex or a canonical, because the crawler never reads it.
Fix errors and redirects
Linking directly to the final destination instead of going through a redirect, returning 404 or 410 for deleted pages and fixing soft 404s frees up resources immediately.
Keep the server healthy
Low response times and rare errors allow the crawler to increase its pace. A spike of 5xx or 429 errors, on the other hand, makes it slow down for days.
Signal changes
A sitemap with an accurate lastmod helps the engine understand which pages have really changed. A lastmod artificially updated on every page every day, by contrast, makes the data unreliable.
How to measure it
The most reliable data comes from server logs: by filtering requests from verified crawlers you can see which sections get visited, how often, and how much of the activity ends up on useless URLs. The crawl statistics in webmaster tools show the trend of requests, response times and the distribution of status codes.
| Indicator | What it suggests |
|---|---|
| Many requests to URLs with parameters | Facets or tracking to contain |
| Lots of 3xx responses | Internal links pointing to redirects |
| Rising response times | Server under strain, crawling will slow down |
| Important pages rarely visited | Weak or overly deep internal links |
In short
- Crawl budget comes from server capacity and the engine's interest.
- It matters for large, dynamic sites or those that generate endless URLs.
- Typical waste comes from facets, parameters, redirects, soft 404s and duplicates.
- A URL blocked in robots.txt cannot pass on a noindex or a canonical.
- Server logs are the most reliable measure of how the budget is spent.
