The robots.txt file: syntax, precedence and mistakes to avoid

Robots.txt is a text file of just a few lines, yet it is one of the most frequent causes of SEO disasters. One slash too many can exclude an entire site from crawling; a badly written rule can leave open what you meant to close. It pays to know its grammar well: it is simpler than it looks, but it has a few traps.

Where it lives and what it is for

The file must sit at the root of the host, that is at the address /robots.txt. It applies only to that host and that protocol: a subdomain needs its own file, and the HTTP and HTTPS versions are treated as separate.

Its job is only one: to tell crawlers which paths not to crawl. It is not a security system, because anyone can read it and malicious crawlers ignore it. Nor is it a tool for removing pages from the index.

Basic syntax

A robots.txt is made of groups. Each group starts with one or more User-agent lines and continues with the rules.

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/public/

User-agent: Googlebot-Image
Disallow: /drafts/

Sitemap: /sitemap.xml

• User-agent states which crawler the group applies to; the asterisk means all of them.
• Disallow excludes paths that begin with the given value.
• Allow reopens a path inside an excluded area.
• Sitemap points to the location of the sitemap and, strictly speaking, should be written as a full URL; it is independent of the groups.

An empty Disallow: means «no restrictions». Disallow: / means «everything excluded». The difference is a single character.

Wildcards

The major engines support two symbols:

• * matches any sequence of characters;
• $ marks the end of the URL.

Useful examples:

• Disallow: /*?sort= blocks URLs with the sorting parameter;
• Disallow: /*.pdf$ blocks files ending in .pdf;
• Disallow: /search also blocks /searching-for-a-home, because matching is by prefix. To limit it to the folder you need /search/.

Who wins when rules contradict each other

A crawler picks only one group: the one with the most specific user-agent that applies to it. If there is a group for Googlebot, Googlebot completely ignores the one with the asterisk. A classic mistake is adding a dedicated group with a single rule and discovering that all the general exclusions no longer apply to that crawler.

Within the group, Google applies the rule with the longest path matching the URL. When lengths are equal, Allow, the less restrictive rule, prevails.

The most common mistakes

Blocking to remove from the index

If a page is already indexed and then blocked, the engine can no longer read it and therefore does not see any noindex. The URL may stay in the results, without a description. To remove a page you first need noindex with crawling allowed, and only afterwards, if necessary, the block.

Blocking CSS and JavaScript

Excluding asset folders prevents the engine from rendering the page as a user sees it. The result can be a wrong assessment of layout and content.

The staging robots.txt that ends up in production

A test environment protected with Disallow: / that gets copied to production during a release blocks the whole site. It is a check to include in every release checklist.

Using unsupported directives

Noindex inside robots.txt is not supported by Google. Crawl-delay is ignored by Google but respected by some other engines. Relying on these lines means getting different behaviour depending on the crawler.

Server errors on the file

If robots.txt returns a 5xx error, crawlers may interpret the situation cautiously and suspend crawling. A 404, on the other hand, is read as «no restrictions».

How to verify it

Before publishing a change it is worth testing it: webmaster tools show which version of the file was read and whether there were problems fetching it, and various libraries and crawlers let you simulate whether a given URL is allowed for a given user-agent. Keeping the file under version control, like the rest of the code, helps you understand when and why it changed.

In short

• Robots.txt controls crawling, not indexing.
• A crawler follows only one group, the most specific for its user-agent.
• Within the group the longest rule wins; on a tie, Allow prevails.
• Do not block CSS and JavaScript needed for rendering.
• Test every change and make sure staging never ends up in production.