Crawl Budget: Who Should Care and Who Should Ignore It
Crawl budget is the number of URLs a search engine will crawl on your site over a given period. It is shaped by two things: how much crawling your server can handle without strain, and how much Google wants to crawl based on the site's perceived value. For most sites it is a non-issue. For a few, it is a real cap on growth.
What crawl budget actually is
Google describes it as the combination of crawl capacity and crawl demand. Capacity is how fast it can crawl without hurting your server. Demand is how much it wants to, driven by popularity, freshness, and quality. The result is a practical limit on how many of your URLs get visited, and how often.
The word budget makes people anxious, which is where a lot of wasted effort comes from. So let us be blunt about who this applies to.
Who should ignore it
If your site has a few thousand URLs or fewer, crawl budget is not your problem. Google can and will crawl a small site thoroughly. Time spent worrying about crawl budget on a 500-page site is time stolen from content, internal linking, or fixing the pages that are genuinely thin. Ignore it and move on.
This is the overrated side of the topic. Plenty of small-site owners rewrite robots.txt and prune pages in the name of crawl budget when the real constraint is that the content does not earn links or rankings. The crawler was never the bottleneck.
Who should care
Crawl budget becomes a real constraint at scale and complexity. The sites that should pay attention share a few traits.
| Site profile | Why crawl budget bites |
|---|---|
| Large ecommerce, 100k+ URLs | Faceted navigation and parameters multiply crawlable URLs fast |
| Marketplaces | User activity generates near-infinite URL combinations |
| Large publishers with deep archives | Millions of old URLs compete for crawl with new content |
| Sites with heavy parameters | Sorting, filtering, and tracking params create endless variants |
Scroll table horizontally →
On these sites, Googlebot can spend most of its requests on low-value URLs while important pages go days or weeks between crawls. New products take too long to appear. Updated content lags. That is a crawl budget problem worth solving.
Signals you have a problem
Do not act on a hunch. Confirm it. The clearest evidence comes from server logs and the Crawl Stats report in Search Console.
- Logs show Googlebot spending a large share of requests on parameter or filter URLs.
- Important pages are crawled rarely, while low-value URLs are crawled often.
- New or updated pages take a long time to be discovered and indexed.
- Crawl Stats shows a high request volume against a low share of useful pages.
If you want the low-tooling way to check this yourself, our guide to log file analysis without a data team walks through it.
Myths that waste time
A few crawl budget beliefs refuse to die, and they cost small teams real effort. The first is that setting a crawl-delay directive helps. For Google it does nothing, because Googlebot ignores crawl-delay entirely and manages its own rate. Adjusting it is a placebo. The second is that submitting URLs more often, through the sitemap or the indexing request, increases your budget. It does not. Discovery and budget are different things, and flooding Google with submissions changes neither.
Another persistent myth is that a bigger sitemap earns more crawling. Sitemaps help Google find URLs; they do not raise the ceiling on how many it chooses to crawl. And the belief that noindex saves crawl budget gets the mechanism backwards, because Google still has to crawl a page to see the noindex tag. If the goal is to stop crawling entirely, that is a robots.txt job, with all the indexing caveats that come with it.
The thread running through all of these is that crawl budget is earned through site health and importance, not conjured with directives. Faster, cleaner responses and a genuinely valuable site raise crawl demand. Fiddling with settings does not. If you catch yourself reaching for a directive, ask first whether the underlying architecture is the real issue.
What to do about it
The goal is to point crawl at what matters. That is mostly about reducing low-value crawlable URLs and strengthening the paths to important ones.
- Control faceted navigation and parameters, keeping the valuable combinations and restricting the rest.
- Consolidate duplicate and thin URLs so crawl is not spread across near-copies.
- Fix redirect chains and remove links to dead URLs so crawl is not wasted following them.
- Improve internal linking so important pages are shallow and well-connected.
Note what is not on that list: obsessively tuning crawl-delay or rewriting robots.txt as a first move. Crawl budget problems are usually architecture problems wearing a technical disguise. The fix is cleaner URL structure and better internal linking more often than it is a directive.
The takeaway: crawl budget is real, but it is a large-site concern. Confirm it in the logs before you act, and treat it as an architecture question rather than a settings tweak. If your site is in the category that genuinely needs this, a focused crawl budget and indexation engagement will find where crawl is leaking and stop it, and our post on faceted navigation covers the most common culprit.