Pagination: long lists that search engines can actually crawl

A blog with hundreds of posts, a shop with thousands of products, a forum with endless threads: sooner or later a list has to be split into pages. Pagination looks like an interface detail, but for a search engine it is often the only route to older or less prominent content. If that route is broken, the posts at the bottom of the archive become invisible.

What pagination is for

Pages 2, 3 and 4 of a category are rarely interesting destinations in search results. Their value lies elsewhere: they contain the links to the individual items. A product listed only on page 37 of a category is discovered and revisited only if the crawler manages to get that far.

Almost all the practical rules follow from this.

Distinct URLs for each page

Each page of the list must have its own stable URL:

• /category/shoes/
• /category/shoes/page/2/ or /category/shoes/?page=2

Both forms work; consistency is what matters. Avoid:

• pagination that relies on a fragment, such as #page=2, because engines generally ignore whatever follows the hash;
• next-page content loaded only via JavaScript without changing the URL;
• the same page reachable both as /page/1/ and as the base URL: page 1 must coincide with the category URL, and /page/1/ should redirect there.

Canonical: each to itself

The most widespread mistake is pointing the canonical of every paginated page to the first one. It seems logical, "there's only one category", but it tells the engine that pages 2 onwards are duplicates of the first. They aren't: they contain different items. The result is that the engine may stop considering them and following their links.

The correct rule:

The legitimate exception is a "view all" page that really contains the entire list and loads in a reasonable time: in that case the paginated pages can point to it as canonical.

Rel next and prev

For years the rel="next" and rel="prev" tags were used to describe the sequence. Google stated some time ago that it no longer uses them as an indexing signal; other engines and some tools may still read them. They do no harm and can stay, but they don't replace normal links between pages.

Real, crawlable links

The crawler follows <a href> links. A button that calls a JavaScript function without a URL is not a link. On each page it helps to have:

• links to the previous and next page;
• links to a few nearby pages and to the first;
• on very large sites, links at wider intervals, so that page 50 isn't fifty clicks away from the first.

The number of items per page matters too: lists of 10 items across 2,000 products produce 200 pages; lists of 48 produce fewer than 50, and every product is closer to the start.

Infinite scroll and "load more"

Infinite scroll is convenient for users, but crawlers don't scroll. If the following items appear only after a scroll event, as far as the engine is concerned the list ends with the first results.

The solution is to build infinite scroll on top of traditional pagination:

1. each block of results corresponds to a page URL, for example ?page=3;
2. when the user scrolls and the next block loads, the URL in the address bar is updated with the History API;
3. the <a href="?page=4"> links are present in the HTML anyway, perhaps shown at the bottom or in a navigation area;
4. opening ?page=3 directly shows the content of that block.

The same applies to a "load more" button: if it is a real link to the next page, enhanced with JavaScript, it works for both users and crawlers.

Noindex on paginated pages?

Putting noindex on pages 2 onwards is something I often see done to "clean up the index". The risk is that, over time, the engine reduces crawling of pages that are permanently excluded and therefore follows their links less. If the aim is to keep them out of results, it usually isn't necessary: they rarely compete with the first page. Better to leave them indexable with distinct titles, such as "Shoes – page 2".

In short

• Pagination exists mainly to let items at the bottom of the list be discovered.
• Each page has its own URL, never a fragment after the hash.
• Each page's canonical points to itself, not to the first page.
• Links between pages must be real, crawlable <a href> elements.
• Infinite scroll and "load more" should be built on top of real paginated URLs.