SEO glossary

Crawl budget

The number of URLs a search engine is willing and able to crawl on a website in a given period.

Crawl budget depends on how fast a server responds and how much demand there is for its pages. It matters mainly for large sites with hundreds of thousands of URLs.

Wasted crawl budget usually comes from endless parameter URLs, redirect chains and low-value pages. robots.txt and clean internal linking help focus crawling on important pages.