The page indexing report: how to read the statuses

The page indexing report in Google's webmaster tools divides known URLs into two groups: indexed and not indexed. The second group alarms almost everyone the first time they open it, because it often contains more URLs than the first. In reality most of those statuses are normal and intended. The work consists of distinguishing what is expected from what is a problem.

How it's organised

For each reason for non-indexing, the report shows the number of URLs involved, the trend over time and a list of examples. The examples are a sample, not the complete list, and the data is updated with a delay of a few days.

The exact wording depends on the interface language; here I use equivalent descriptions.

Statuses that are normal, if consistent with your choices

Page with redirect

The URL redirects elsewhere. It's normal for old URLs after a migration. It only becomes a problem if pages that should be indexed appear among them, or if internal links still point to redirected URLs.

Excluded by noindex tag

The page has a noindex. Check the list: if it contains only pages you excluded on purpose, such as cart, private area or internal search, that's fine. If important pages show up, a noindex has slipped through, often from a template or from staging.

Alternate page with proper canonical

The URL declares a canonical pointing to another page and the engine respects it. Typical for parameter variants or printable versions. Normal.

Not found (404)

URLs that return 404. If they are pages deleted on purpose or URLs that never existed, it's normal. Only check the ones that receive internal links or correspond to content that is still live.

Blocked by robots.txt

URLs excluded from crawling. That's fine if the block is intended. Remember, though, that a blocked URL can still appear in the index, without content, if it receives links: to truly exclude it you need a noindex on a crawlable page.

Statuses that need attention

Discovered, not yet crawled

The engine knows the URL but hasn't visited it yet. Typical causes:

• the site has more URLs than the engine considers worth crawling, often because of parameters and filters;
• the server is slow or has returned errors, and the engine has slowed down;
• the pages receive few internal links and look unimportant.

If the status concerns a few new pages, just wait. If it has concerned an entire section for weeks, you need to work on internal links, reducing pointless URLs and server performance.

Crawled, not indexed

The engine visited the page and decided not to include it, at least for now. It is almost always a matter of perceived value, not a technical issue:

• content very similar to other pages on the same site;
• thin pages, with little original text;
• pages generated automatically in large numbers, such as tags with a single post.

The answer isn't to submit the page repeatedly, but to improve it, merge it with others or remove it.

Duplicate, engine chose a different canonical from yours

You indicated a canonical, but the engine chose another. This happens when other signals contradict your indication: internal links pointing to the alternative version, a sitemap with the "wrong" URL, inconsistent redirects, near-identical content. Align all signals towards the URL you prefer.

Duplicate without user-selected canonical

The engine found duplicate pages and none declares which is the main one. Add a self-referencing canonical to the main pages and indicate the preferred version on the duplicates.

Soft 404

The page returns 200 but looks empty or like an error. Fix the status code or add real content.

Server error (5xx) and redirect problems

Genuine technical errors: they should be fixed as a priority, because they may affect important pages.

A working method

In my work I tackle the report in this order:

1. I start from the sitemap: I filter the report on URLs submitted in the sitemap, because those are the ones I declared important. Anything there that isn't indexed deserves an explanation;
2. I look for anomalies in the chart: a sudden spike in a status almost always points to a specific release or change;
3. I check the examples: I open a few URLs with the URL inspection tool to see how the engine saw them;
4. I fix and validate: after the fix, the validation feature asks the engine to recheck the URLs in that group.

Ignoring a high number of non-indexed URLs is often the right choice, as long as you know why they aren't indexed.

In short

• Many non-indexing statuses are normal and consistent with your choices.
• "Discovered, not crawled" points to priority, internal linking or server problems.
• "Crawled, not indexed" is almost always about the value of the content.
• If the engine picks another canonical, align links, sitemap and redirects.
• Start from the sitemap URLs and investigate spikes in the chart.