When people talk about duplicate content they think of text copied from other sites. But the most widespread duplication is internal and purely technical: the same page responds at several addresses that differ by a detail. It is not a penalty, but it scatters signals, consumes crawling and forces the engine to guess which version to show.
The most common variants
Take any page. It might be reachable in all these ways:
http://example.com/about-us
https://example.com/about-us
https://www.example.com/about-us
https://www.example.com/about-us/
https://www.example.com/About-Us/
https://www.example.com/about-us/index.html
https://www.example.com/about-us/?ref=footer
Seven addresses, one page. If they all return 200, to a search engine they are potentially seven different documents.
Protocol
If the site responds on both HTTP and HTTPS without redirecting, there are two complete copies of the site. After moving to HTTPS, the HTTP version must permanently redirect on every URL, not just the home page.
Host with and without www
You need to choose a main host and redirect the other one. It does not matter which, what matters is that there is only one and that internal links, canonicals and sitemap always use it.
Trailing slash
/about-us and /about-us/ are, as far as HTTP is concerned, two distinct URLs. Many servers treat them as equivalent and return 200 for both. You need a convention, enforced with a redirect from one form to the other.
Upper and lower case
The path of a URL is case-sensitive. On servers that are not, such as some Windows configurations, /About-Us/ and /about-us/ return the same page. The usual solution is to force lower case with a redirect.
Index files
/about-us/ and /about-us/index.html or index.php show the same content. The index file name should never appear in links and should redirect to the folder.
Parameters
Tracking, session or referral parameters generate endless copies. A self-referencing canonical largely neutralises them; better still is not to put them in internal links.
Other frequent duplications
- Multiple paths to the same resource: a product under
/women/shoes/modeland/sale/model. - Printable versions or «AMP» versions no longer maintained.
- Test environments publicly accessible that replicate production.
- Archive pages by author, date and tag that repeat the same excerpts.
- Secondary domains or old domains serving the same site without redirects.
How to find them
Manual test
For each variant in the initial list, request the URL and look at the status code. The correct behaviour is a 301 to the canonical version, in a single hop. A command-line tool such as curl shows the headers instantly:
curl -sI http://example.com/about-us
Crawler
A desktop crawler starting from the home page follows only real links; if it finds variants, it means the site itself generates them in internal links. This is the most serious case, because you are actively signalling the wrong version.
Webmaster tools
Indexing reports list pages considered duplicates, distinguishing between «duplicate without user-selected canonical» and «Google chose a different canonical». Both entries deserve attention.
Server logs
Crawler requests to non-canonical variants show how much crawl activity is being dispersed.
How to unify them
- Decide on the canonical form: protocol, host, trailing slash, lower case.
- Apply 301 redirects at server or CDN level for all other forms, in a single hop. Avoid chains such as HTTP → HTTPS → www → trailing slash.
- Align internal links: menus, footer, content and breadcrumbs must use the canonical form.
- Set a self-referencing canonical on every page, with an absolute URL in the correct form.
- Regenerate the sitemap with canonical URLs only.
- Verify after release with the crawler and with direct requests to the variants.
| Variant | Action |
|---|---|
| HTTP | 301 to HTTPS |
| secondary host | 301 to main host |
| non-conforming trailing slash | 301 to the chosen convention |
| upper case | 301 to lower case |
| index.html | 301 to the folder |
| tracking parameters | canonical to the clean URL |
In short
- The same page can respond at many URLs that differ by a detail.
- Protocol, host, trailing slash, case and index files should be unified with redirects.
- The redirect must reach the canonical form in a single hop.
- Internal links must always use the canonical version.
- A self-referencing canonical and a clean sitemap complete the job.
