You do not need a data warehouse to read server logs. For most sites, two to four weeks of raw access logs, a spreadsheet or a few command-line tools, and an afternoon will tell you more about how Googlebot treats your site than any crawl simulation can.
Why logs beat a crawl
A crawler like Screaming Frog shows you what could be crawled. Logs show what was crawled, by whom, how often, and what response it got. That is the difference between a model and the truth. Only the logs tell you that Googlebot spends 40% of its requests on parameter URLs nobody searches for, or that it keeps hitting a set of pages you deleted six months ago.
This is the single most underrated technique in technical SEO. It is also the one people skip, because getting the files feels like a chore. It is worth the chore.
Getting the log data
Ask for raw access logs from the web server, or from the CDN if you sit behind Cloudflare, Fastly, or Akamai. You want the request line, status code, user agent, timestamp, and ideally the response size. Two to four weeks is enough to see patterns without drowning in data. If the site is huge, a representative sample by section works.
One caution: user agents are easy to spoof. Verify Googlebot by reverse DNS lookup rather than trusting the string, or you will count fake bots as real crawl.
Processing it without a warehouse
A spreadsheet handles a few hundred thousand rows. Beyond that, the command line is your friend, and you do not need to be an engineer to use it. Here is a starting point for finding the most-crawled URLs from an access log.
# Count Googlebot requests per URL, top 30
grep -i "googlebot" access.log \
| awk '{print $7}' \
| sort | uniq -c \
| sort -rn | head -30That one command tells you where crawl is going. Swap the field number to match your log format. From there you can filter by status code, group by URL pattern, or pull the results into a spreadsheet for charts.
What to read first
Do not try to read everything. Look at four things, in order.
| Signal | Question it answers | Why it matters |
|---|---|---|
| Requests by URL pattern | Where is crawl actually spent? | Reveals crawl waste on facets and parameters |
| Status code mix | How many 404s, 301s, 5xx is Google hitting? | Chains and errors waste budget and signal poor health |
| Crawl frequency by template | How often are money pages crawled? | Rarely-crawled key pages rank and update slowly |
| Orphan hits | Which crawled URLs have no internal links? | Shows leftover or generated URLs Google still finds |
Scroll table horizontally →
What to act on
The point of the analysis is a short list of changes, not a chart. Typical actions that come out of a first log review:
- Block or canonicalise the parameter and facet patterns eating crawl, carefully, protecting any that rank.
- Fix redirect chains Google keeps following, collapsing them to single hops.
- Add internal links to important pages that are crawled rarely.
- Investigate 5xx spikes that suggest server strain under crawl.
If controlling crawl on facets is where this leads, our post on faceted navigation and SEO covers how to do it without losing the filtered pages that earn traffic.
Pitfalls to avoid
A few things trip people up on their first log analysis, and knowing them in advance saves a wasted afternoon. The first is trusting the user agent string. Anyone can send a request claiming to be Googlebot, and plenty of scrapers do, so counting unverified user agents inflates your numbers with traffic that has nothing to do with search. Always verify by reverse DNS, or by matching against Google’s published IP ranges, before you draw conclusions.
The second is analysing too short a window. A single day is noise. Crawl patterns move over weeks, and a big template only shows its true crawl share across a proper sample. Two to four weeks is the sweet spot for most sites. The third is mixing CDN logs and origin logs without realising it, which double counts requests that the CDN served from cache versus passed through. Decide which layer you are measuring and stick to it.
Finally, do not treat every crawled 404 as an emergency. Some 404s are correct and healthy: pages you deliberately removed should return 404 or 410, and Google will keep checking them for a while. The signal that matters is a rising volume of 404s on URLs that used to be valuable, or crawl being spent heavily on error responses instead of live pages. Context turns a scary-looking number into a decision.
Making it a habit
A one-off log review finds the obvious waste. The value compounds when you repeat it. Run it quarterly on a large site, or after any migration, and compare against the last run. Crawl patterns drift as templates change, and problems reappear.
The takeaway: logs are the closest thing to watching Googlebot over its shoulder, and you can start with tools you already have. If the site is large enough that a spreadsheet chokes and the patterns get complicated, that is the point where a dedicated crawl and indexation engagement pays for itself, because the analysis becomes continuous rather than a one-time snapshot.