When a page does not show up in search results, the right question is not «why is Google penalising me?» but «at which stage did it stop?». A search engine works in distinct steps: first it discovers the address, then it downloads it, then it interprets it and finally it decides whether to keep it in the index. Each step has its typical failures, and understanding the sequence is the fastest way to diagnose a problem.
Stage 1: discovering URLs
Before anything else, the engine has to know that an address exists. There are three main sources:
- links from other pages, internal or external, that the crawler has already visited;
- XML sitemaps declared in the robots.txt file or submitted through webmaster tools;
- direct notifications, such as manually submitting a URL or notification protocols supported by some engines.
A page that receives no links and is not listed in a sitemap is, for all practical purposes, invisible. These are the so-called orphan pages: they exist on the server, but nobody announces them. In my work I often find them after a redesign, when a menu is rebuilt and entire sections remain reachable only by typing the address.
Stage 2: crawling
Once the URL is discovered, the crawler queues it and, when its turn comes, makes an HTTP request. Before doing so it checks the host's robots.txt file: if the path is excluded with Disallow, the request is never made.
During crawling, what matters most is:
- the status code returned (200, 301, 404, 500 and so on);
- the server's response speed, because a slow or error-prone host makes the crawler back off;
- the HTTP headers, which may contain instructions such as
X-Robots-Tagor a canonical.
A frequent mistake is believing that blocking a page in robots.txt removes it from the results. It does not: the block prevents the page from being read, not its address from being known. If other pages link to it, the URL can still appear, perhaps without a description.
Stage 3: rendering
Many modern sites build their content with JavaScript. In that case, the HTML downloaded in the previous stage is little more than an empty shell. To see the page as a user sees it, the engine has to run the scripts in a headless browser.
Rendering is expensive, so it may happen later than crawling. The practical consequences:
- text and links that exist only after JavaScript runs may be discovered late;
- if a script fails, or a required resource is blocked by robots.txt, the rendered page may be incomplete;
- links generated by click events, without a real
<a href>element, are usually not followed.
That is why, wherever possible, the main content and navigation links should already be present in the initial HTML.
Stage 4: indexing
With the rendered page in hand, the engine analyses text, headings, images, structured data and technical signals. Two important decisions are made here.
Canonicalisation
If several URLs show the same content, the engine picks one as the representative, the canonical, and groups the others as duplicates. The rel="canonical" tag is a strong hint, but it is weighed against other clues: redirects, internal links, inclusion in the sitemap, HTTPS.
Inclusion or exclusion
Not everything that is crawled gets indexed. A page can be excluded because:
- it contains a
noindexin the meta robots tag or in the headers; - it is considered a duplicate of another page;
- it has little useful content or looks like an error page in disguise (the so-called soft 404);
- the engine simply judges it not useful enough to keep.
Indexing is not a right: it is the engine's choice, and with limited resources it prefers to spend them on pages it considers valuable.
A table for diagnosis
| Symptom | Likely stage | First check |
|---|---|---|
| The URL has never been visited | Discovery | Internal links and sitemap |
| Visited but «blocked» | Crawling | Rules in robots.txt |
| Content missing from the rendered preview | Rendering | JavaScript errors, blocked resources |
| Crawled but not indexed | Indexing | Quality, duplicates, noindex |
| Indexed under a different URL | Canonicalisation | Canonical, redirects, internal links |
Tools to find where things stopped
The webmaster tools of the major engines offer a URL inspection feature that shows whether the page is known, when it was crawled, which canonical was chosen and how it looks after rendering. Alongside that:
- a desktop crawler that simulates crawling and reports status codes, canonicals and depth;
- the server logs, which tell you with certainty which URLs the crawler requested and with what result;
- a comparison between the HTML source and the rendered DOM, to see what depends on JavaScript.
In short
- Search works in four steps: discovery, crawling, rendering, indexing.
- Blocking in robots.txt prevents reading, not the appearance of the URL.
- Essential content and links should already exist in the initial HTML.
- Being crawled does not guarantee being indexed: quality and uniqueness matter.
- Diagnosing stage by stage avoids looking for the problem in the wrong place.
