Service · Crawl budget & indexation
Crawl budget and indexation work for sites too large to crawl casually
Crawl budget is the number of URLs a search engine will crawl on your site in a given period, shaped by how important and how healthy it thinks the site is. Indexation is the separate question of which crawled pages actually make it into the index. On large sites these two problems quietly cap growth. We diagnose them from log data and fix them without breaking the pages that already rank.
This work pays off on large, complex sites
If you run a few hundred pages, crawl budget is not your problem and anyone telling you otherwise is selling something. This service is for sites where scale itself is the issue: ecommerce catalogues with hundreds of thousands of SKUs, marketplaces generating URLs from user activity, publishers with deep archives, and enterprise sites spanning many regions.
At that scale, Googlebot cannot and will not crawl everything often. The job is to make sure it spends its attention on the URLs that earn revenue, and stops wasting it on infinite parameter combinations and filtered views nobody searches for.
Robots directives, used with care
User-agent: *
# Block genuine crawl traps, not useful pages
Disallow: /*?sort=
Disallow: /*?sessionid=
Disallow: /search/
# Keep valuable filtered URLs crawlable
Allow: /shoes/waterproof/
Sitemap: https://example.com/sitemap_index.xmlRobots.txt controls crawling, not indexing. A blocked URL can still be indexed if it is linked. We pair crawl rules with canonical and noindex decisions so the two do not contradict each other.
Reading the signals
What the Search Console Pages report is telling you
These four statuses account for most indexation questions on large sites. Each points in a different direction, so the fix depends on reading them correctly.
| GSC status | What it usually means | Direction of fix |
|---|---|---|
| Discovered, currently not indexed | Google knows the URL but has not crawled it, often a crawl-priority or quality signal | Improve internal links and site value signals to the URL |
| Crawled, currently not indexed | Crawled and judged not worth indexing, usually thin or duplicate | Strengthen content, consolidate duplicates, add internal links |
| Duplicate without user-selected canonical | Google chose a different canonical than you intended | Fix canonical signals and internal linking to the preferred URL |
| Indexed, not submitted in sitemap | URLs indexed that you did not list, often bloat from facets or parameters | Decide whether to keep, canonicalise, or block the pattern |
Scroll table horizontally →
Scope
What the engagement covers
Log file analysis
We process server or CDN logs to see real Googlebot behaviour: crawl frequency by template, wasted requests, orphan hits, and status code issues.
Index bloat cleanup
We find the thin and duplicate URLs sitting in the index and consolidate them with canonicals, noindex, or removal, protecting anything valuable first.
Faceted navigation control
We decide which filter combinations deserve indexing and crawling, and how to stop the rest without hiding pages that rank.
Internal linking at scale
We improve how link equity and crawl priority flow, reducing click depth to important pages and fixing orphan pages through real internal links.
If you want to try the log-file side yourself first, our guide on running a log file analysis without a data team covers the low-tooling version of what we do.
FAQ
Crawl and indexation questions
Do I even have a crawl budget problem?
Probably not, if your site is under a few thousand URLs. Crawl budget becomes a real constraint on large sites, hundreds of thousands of URLs and up, especially with faceted navigation or heavy parameters. The signal is Googlebot spending its requests on low-value URLs while important pages go days or weeks between crawls. We confirm it in the logs before we act on it.
What is index bloat and why does it matter?
Index bloat is when a large number of thin, duplicate, or low-value URLs end up indexed. It dilutes the signals that should concentrate on your important pages, wastes crawl on pages that will never rank, and makes reporting noisy. On large ecommerce sites it is one of the most common and most fixable problems.
How does log file analysis help?
Server logs show what Googlebot actually requested, how often, and what it received. That is ground truth, unlike a crawl simulation. Logs reveal crawl waste on parameters and facets, orphan pages Google still hits, status code problems, and which templates get crawled most, which tells you where to focus.
Will controlling crawl hurt pages that currently rank?
Not if it is done carefully. The risk is blocking or de-indexing URLs that quietly earn traffic. We identify the valuable filtered and parameter URLs first, protect them, and only then restrict the genuine dead ends. Blunt robots.txt rules applied without that step are how people lose rankings by accident.
How is internal linking part of this?
Internal linking is how crawl priority and link equity flow through a site. Pages buried many clicks deep with few internal links get crawled rarely and rank poorly. Fixing internal linking at scale, through hubs, related links, and cleaner architecture, is often more effective than any robots.txt change.
Related reading
Crawl and index work is the core of what a technical SEO agency does on large sites. It is unglamorous, it is measurable, and it is where the biggest quiet gains usually sit.
Start here
Large site where the wrong pages get crawled?
Send the domain and rough URL count. If you can share log access, even better, and the first read will be far more specific.