XML sitemaps: structure, limits and what to actually include

An XML sitemap is a list of addresses that the site owner hands to search engines, saying: «these are the pages I consider important». It does not force anyone to index them, but it speeds up discovery and provides a valuable point of comparison for diagnosis. Done badly, it becomes noise that the engine learns to ignore.

The format

A sitemap is an XML file with a root element urlset and one url element for each page:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/services/</loc>
    <lastmod>2026-03-14</lastmod>
  </url>
</urlset>

The protocol defines four elements:

• loc, required: the absolute URL of the page;
• lastmod: the date of the last significant change;
• changefreq and priority: update frequency and relative priority.

Google has stated that it ignores changefreq and priority, while it uses lastmod when it considers it reliable. In practice it makes sense to focus on loc and lastmod and not waste time on the other two.

The limits

Each file can contain at most 50,000 URLs and weigh at most 50 MB uncompressed. The file may be served gzip-compressed, but the size limit refers to the decompressed content. Beyond these thresholds the URLs are split across several files, gathered by a sitemap index.

Other constraints to respect:

• the file must be UTF-8 encoded;
• URLs must be absolute, with special characters correctly escaped, for example & written as &;
• a sitemap should list URLs from the same host it lives on, except for cross-verification setups.

What to include

The rule is simple: only canonical, indexable URLs that return 200. Every address in the sitemap should be a page you want to see in the results.

So you should exclude:

• URLs that redirect;
• pages with noindex;
• pages blocked in robots.txt;
• parameter variants, duplicates, non-canonical versions;
• error pages, including those disguised as 200;
• internal search results and utility pages.

A «dirty» sitemap creates a paradox: you declare as important pages that you are simultaneously telling the engine not to index. The engine receives contradictory signals and tends to trust the whole file less.

Using lastmod honestly

lastmod is useful only if it reflects real changes to the main content. Common mistakes:

• setting the sitemap generation date on every URL;
• updating the date when only a sidebar widget, the footer copyright or a counter changes;
• leaving old dates on pages that have been completely rewritten.

The outcome is the same: the engine stops taking the field into account. It is better to derive the value from the content's modification date in the database or source file.

Generating and maintaining it

A handwritten sitemap is acceptable only for tiny, stable sites. In every other case it should be generated automatically:

• the most widespread CMSs produce it natively or through extensions;
• static site generators create it at build time;
• custom applications can build it from a database query, filtering published and indexable pages.

What matters is that the generation logic follows the same rules as the site: if a page has noindex or a different canonical, it must not end up in the list.

How to submit it

There are two complementary ways:

1. add the Sitemap: line with the full URL to robots.txt, which makes it discoverable by any crawler;
2. submit it in the engines' webmaster tools, which in return provide a report on discovered URLs, errors and indexing status.

The second step is the most useful for diagnosis: by comparing the submitted URLs with those actually indexed you immediately see which sections have problems.

A periodic check

A desktop crawler can read the sitemap and automatically check every listed URL, making the job quick even across thousands of pages.

In short

• The sitemap helps discovery; it does not guarantee indexing.
• At most 50,000 URLs and 50 MB uncompressed per file.
• Include only canonical, indexable URLs that return 200.
• lastmod works only if it reflects real changes.
• Declare it in robots.txt and submit it in webmaster tools.