Meta robots and X-Robots-Tag: noindex, nofollow and other rules

If robots.txt decides what the crawler may read, robots directives decide what the engine may do with what it has read. They are the right tool for keeping a page out of the index, limiting the snippet in the results or preventing links from being followed. They come in two forms: a tag in the HTML and an HTTP header.

The two forms

The meta tag

It goes in the page's <head> section:

<meta name="robots" content="noindex, follow">

The value robots addresses all crawlers. You can target a specific engine by replacing it with the name of its crawler, for example googlebot.

The HTTP header

The same instruction can travel in the server's response:

X-Robots-Tag: noindex

The header is essential for files that have no <head>: PDFs, images, downloadable documents, feeds. It is also handy for applying a rule to whole groups of files through server configuration, without touching the page code.

The main directives

The values index and follow are the default behaviour: writing them does no harm, but adds nothing.

There is also the data-nosnippet attribute, which can be applied to individual HTML elements to exclude only a portion of text from the snippet, such as a repeated notice or a piece of data you do not want shown in previews.

The golden rule: the crawler must be able to read

A robots directive works only if the engine downloads the page. If the URL is blocked in robots.txt, the noindex is never seen. This is the most widespread confusion: people block and add noindex «just to be safe», achieving the opposite of what they wanted.

The correct sequence for removing an existing page from the index is:

1. keep crawling allowed;
2. add noindex;
3. wait for the engine to come back and remove it;
4. only then, if you need to save crawl activity, consider a block.

When instructions contradict each other

If a page contains several directives, for example a meta tag and a header, or two meta tags generated by different plugins, engines tend to apply the most restrictive one. An index in the theme and a noindex added by a plugin result in noindex.

Other typical conflicts:

• noindex and canonical together: saying «do not index me» and «I am the main version» or «I am a duplicate of this other page» sends mixed signals. Better to pick just one of the two tools depending on the goal;
• noindex on pages listed in the sitemap: the sitemap should list only URLs you want indexed;
• noindex on heavily linked internal pages: if a page is excluded but receives many links, perhaps the structure needs rethinking.

Concrete use cases

• Internal search results: usually noindex, because they generate endless low-value combinations.
• Thank-you pages after a purchase or a form: noindex, they are of no use to searchers.
• PDF documents duplicating HTML pages: an X-Robots-Tag: noindex header so that the web version surfaces.
• Archive pages with thin content, such as tags with a single article: noindex until they grow.
• Time-limited offers: unavailable_after prevents them from remaining visible past the date.

A configuration example for PDFs

On an Apache server, for example, you can apply the header to all PDFs in a folder:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

The equivalent exists in Nginx and in most CDNs. After the change it is worth checking the response with a tool that shows headers, because a syntax error here is silent.

How to check what the engine sees

The HTML source is not enough. You need to verify:

• the rendered DOM, because a noindex can be added or removed by JavaScript;
• the HTTP headers of the final response, after any redirects;
• the URL inspection report in webmaster tools, which states whether the exclusion is due to noindex.

A desktop crawler configured to read both meta tags and headers lets you run the same check across the whole site at once.

In short

• Meta robots applies to HTML pages, X-Robots-Tag to any file.
• Noindex works only if the page can be crawled.
• With conflicting directives, the most restrictive one wins.
• Do not list pages marked noindex in the sitemap.
• Check the rendered DOM and headers, not just the source.