Server logs for SEO: what crawlers really do on your site

Webmaster tools show a summary of what the engine thinks of a site. Server logs show what the crawler actually did, request by request: which URLs it asked for, when, and with what response. They are the most direct source for understanding whether important pages get visited and whether the crawler's time is being wasted elsewhere.

What a log line contains

In the most common format, the "combined" format used by Apache and Nginx, a line looks like this:

66.249.66.1 - - [12/Mar/2026:10:15:32 +0100] "GET /products/red-shoes HTTP/1.1" 200 18234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1)"

The useful elements are:

If the site sits behind a CDN, the origin server's logs may show the CDN's IP instead of the client's: in that case you need the CDN's logs or a header carrying the original IP.

The user agent isn't enough

Anyone can write "Googlebot" in the user agent. Scrapers and automated tools often do, precisely to avoid being blocked. Before drawing conclusions you have to separate real crawlers from fake ones.

The reliable method is the double DNS check:

1. run a reverse DNS lookup on the IP: for a genuine Googlebot the returned name ends in googlebot.com or google.com;
2. run a forward lookup on that name: it must return the same IP you started with.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The second step is essential, because whoever controls an IP address can set its reverse DNS to anything they like. Bing follows the same pattern with names ending in search.msn.com. Several engines also publish the list of their IP ranges, useful for automated checks on large volumes.

The questions to answer

Once requests from verified crawlers have been filtered, the analysis becomes a series of concrete questions.

Are the important pages visited?

Compare the list of URLs in the sitemap with those requested by the crawler over the last thirty days. Pages never visited are candidates for a discovery problem: missing internal links, excessive depth, unintended exclusions.

Where is crawling concentrated?

Group requests by site section or URL pattern. If a large share goes to filter parameters, internal search pages or infinite calendars, the crawler is spending time on low-value URLs.

Which responses does it get?

How often does it come back?

A page updated often but visited rarely suggests the engine isn't noticing those changes. A page never modified but visited every day may suggest the crawler finds little new elsewhere.

What happens after a change?

After a migration, a release or a robots.txt change, logs show within a few days whether crawler behaviour has changed, long before organic traffic reveals it.

Tools

For small sites, command-line tools are enough:

grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn

This command counts responses by status code among requests claiming to be Googlebot, to be refined afterwards with IP verification. For larger volumes it's better to load the logs into a database or use a log analyser built for SEO, which cross-references the data with a site crawl.

In my work I keep at least a few months of logs, compressed and archived: many questions, such as "when did it stop visiting this section?", require history.

Mind the personal data

Logs contain IP addresses and sometimes parameters with user data. SEO analysis only needs crawler requests: it's best to filter them at source, restrict access to the raw files and decide how long to keep them.

In short

• Logs show what crawlers actually do, not what you assume.
• The user agent is easily faked: verify IPs with reverse and forward DNS.
• Compare visited URLs with the sitemap to find neglected pages.
• Group by section and status code to see where crawling is wasted.
• Keep a history, filtering and protecting personal data.