Sooner or later every site deletes pages: discontinued products, past events, outdated articles, services no longer offered. There's nothing wrong with that: a healthy site renews itself. The problem arises when the removal is communicated the wrong way, to the engine or to the user. The HTTP status code is how the server says what happened to that page, and choosing the right one avoids both wasted crawling and frustrating experiences.
The codes involved
| Code | Meaning | Typical use |
|---|---|---|
| 200 | the page exists | normal content |
| 301 | moved permanently | an equivalent replacement exists |
| 404 | not found | page removed or never existed |
| 410 | gone | intentional, permanent removal |
404 and 410
Both say the content isn't there. The difference is one of intent: a 404 doesn't specify whether the absence is temporary or permanent, while a 410 states that the page was removed on purpose and won't come back. In practice engines treat the two codes very similarly; a 410 may lead to slightly faster removal from the index.
My rule of thumb: I use 410 when I have consciously decided to remove a group of pages, for example old pages generated by mistake, and 404 as the normal response for anything that doesn't exist.
Having 404s is not a problem in itself. A site with many nonexistent URLs requested from outside, due to broken links or automated attempts, is not penalised for it. It only becomes a problem when the 404s concern pages that should exist or that receive links and traffic.
Soft 404s
A soft 404 is a page that returns 200 but in fact has no content: an empty page, a "product unavailable" message with no other text, an internal search with no results, a redirect of every deleted URL to the home page.
Engines recognise these situations and classify them as soft 404s, but in the meantime they have crawled them as valid pages. The consequences:
- crawl resources spent on useless pages;
- the risk of empty pages ending up in the index;
- confusing data in reports, where the same problem appears under different labels.
The fix is to have the server respond with the correct code: if the content isn't there, 404 or 410, even if the page shown to the user is well designed and useful.
When to redirect
A 301 redirect makes sense when a real replacement exists:
- a product replaced by the new model;
- an article merged into another, more complete one;
- a service page moved to a new address.
It makes no sense to redirect everything to the home page or a generic category. Users looking for a specific product end up on a page that doesn't answer their question, and engines tend to treat these redirects as soft 404s.
Out-of-stock products
For an online shop it helps to distinguish:
- temporarily out of stock: the page stays at 200, with a clear indication of availability, an expected date if known, the option to get notified and alternative products. Structured data must report the real availability;
- discontinued, with a successor: 301 redirect to the new model;
- discontinued, no successor but with traffic: you can keep the page as an information sheet, with links to similar products, if it is useful to users;
- discontinued, no value: 404 or 410.
An error page that helps
The status code is for machines; the page is for people. A good 404 page:
- states clearly that the page doesn't exist;
- keeps the site's menu and look;
- offers a search box and links to the main sections;
- doesn't automatically redirect elsewhere after a few seconds.
Check that it really returns 404: some CMSs serve the error page with a 200 code. It takes a moment to verify:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/nonexistent-page-xyz
Finding the 404s that matter
- Crawler: reports internal links pointing to nonexistent pages; these should be fixed at the source.
- Server logs: show which nonexistent URLs are requested most often and by whom.
- Webmaster tools: list the not-found URLs the engine knows about, often from external links.
- Backlink analysis tools: show which deleted pages still receive links, candidates for a redirect.
In short
- 404 and 410 indicate missing content; 410 declares an intentional removal.
- 404s on URLs without value are normal and don't penalise the site.
- A soft 404 returns 200 with no content: fix it with the right code.
- Redirect only to a real replacement, never everything to the home page.
- The error page must be useful and really respond with 404.
