IndexNow and engine notifications: reporting changes right away

The classic search model is based on waiting: the engine periodically comes back to the site, discovers what's new and processes it. For a site that publishes little, that's perfectly fine. For a site that changes often, with prices, availability, news or listings that expire, the delay between a change and the index update can be a problem. IndexNow is an open protocol created to reduce that delay: instead of waiting for the crawler, the site notifies the engines that a URL has changed.

How it works

The principle is simple:

1. the site generates a key, a random string;
2. it publishes a text file with that key on its own domain, to prove it controls it;
3. whenever a URL is created, modified or deleted, it sends a request to an engine that supports the protocol, giving the URL and the key;
4. the engine verifies the key and queues the URL for a new crawl.

Participating engines share the notifications they receive with each other, so sending them to just one is enough. The protocol was introduced by Bing and Yandex and is adopted by other engines; Google, at the time of writing, does not use it and continues to rely on crawling, sitemaps and its own tools.

Setting it up

The key

The key is a string of hexadecimal characters and dashes, between 8 and 128 characters long. It is published in a text file containing only the key itself, usually at the site root, with a file name equal to the key:

/3f8a2c9e7b1d4f60a5e2c7b9d1f3a8e4.txt

The file can also live at another path, but in that case the key is valid only for URLs under that path, and its location must be stated in the requests.

Notifying a single URL

For one URL, a GET request to the engine's endpoint with the url and key parameters is enough.

Notifying several URLs

For groups of URLs you use a POST request with a JSON body:

{
  "host": "example.com",
  "key": "3f8a2c9e7b1d4f60a5e2c7b9d1f3a8e4",
  "keyLocation": "https://example.com/3f8a2c9e7b1d4f60a5e2c7b9d1f3a8e4.txt",
  "urlList": [
    "https://example.com/products/red-shoes",
    "https://example.com/products/blue-shoes"
  ]
}

The protocol allows up to 10,000 URLs per request, all from the same host.

Responses

A 200 or 202 code means the request was accepted; 403 signals an invalid or missing key; 422 indicates URLs that don't belong to the host or a key that doesn't match; 429 asks you to slow down. Accepted doesn't mean indexed: the engine still decides whether and when to crawl.

When it's worth it

IndexNow is useful when:

• the site publishes or modifies a lot of content every day;
• the information has an expiry date, such as events, offers or listings;
• pages are deleted and you want them to disappear from results quickly;
• a significant share of the audience uses engines that support the protocol.

Many CMSs and some CDNs already offer plugins or settings that send notifications automatically on every publication. In a custom system, you just hook the call to the save event or to the end of the build.

Mistakes to avoid

• Notifying unchanged URLs: sending the whole site on every build speeds nothing up and may lead the engine to give notifications less weight. Only send URLs that were actually created, modified or removed.
• Notifying non-canonical URLs: parameter variants, URLs that redirect or pages with noindex.
• Mishandling the key: the key isn't a secret in the strict sense, since it's public in the file, but it should be generated randomly and not reused across different sites.
• Forgetting the key file after a migration or a hosting change: all later notifications would be rejected.

In my work I log every submission with the date, URL and response code: when something doesn't get updated, the log immediately shows whether the notification went out and how it was received.

What it doesn't replace

IndexNow speeds up the discovery of changes, but it doesn't change the underlying rules:

• a weak page isn't indexed just because it was notified;
• the sitemap remains necessary, including for engines that don't use the protocol;
• internal links remain the main way engines understand the importance of pages;
• canonical, noindex and robots.txt continue to apply exactly as before.

In short

• IndexNow lets the site report new, modified or deleted URLs immediately.
• It's set up with a random key published in a text file on the domain.
• A notification to one participating engine is shared with the others; Google doesn't use it.
• Send only canonical URLs that have really changed, and log the responses.
• It doesn't replace sitemaps, internal links and content quality.