Glossary
What is Crawl Budget?
Crawl budget is the number of pages a search engine's crawler will fetch from a given site within a given timeframe, shaped by the site's perceived importance, size, and server capacity.
Every search engine allocates a finite amount of crawling attention across the entire web, and how much of it lands on any one site depends on two things Google names directly: crawl rate limit (how much crawling the site's server can handle without being overloaded) and crawl demand (how much Google actually wants to crawl the site, based on its perceived popularity and how stale the existing index of it is).
Crawl budget mostly matters for large sites — think tens of thousands of pages or more, or sites that generate pages dynamically (faceted e-commerce filters, auto-generated tag pages). On a site that size, wasting crawl budget on low-value URLs — duplicate parameter variations, thin auto-generated pages, redirect chains — can mean genuinely important new or updated pages wait longer to be crawled and reflected in search results. For a small business site with a few dozen pages, crawl budget is rarely the bottleneck; there's more than enough allocated attention to reach everything already.
Common crawl budget fixes include consolidating or noindexing low-value duplicate pages, fixing redirect chains, blocking crawl traps in robots.txt, and improving internal linking so crawlers discover important pages through fewer hops.