Crawl budget is how many pages a search engine is willing and able to crawl on your site in a given time.
When it matters
- Large sites with tens of thousands of pages.
- Sites that generate endless URLs from filters, calendars or search.
- Sites with lots of broken links, redirects or slow responses.
When it does not
A site with a few hundred pages that are linked well is usually crawled fully.
Signs of a crawl budget problem
- New pages take weeks to be found.
- Important pages are crawled rarely while filter or calendar pages are crawled often.
- Server logs show crawlers spending most of their time on low-value URLs.
Looking at server logs
Server access logs show every request from crawlers: which URLs, how often and with which status codes. They are the most direct way to see where crawl time goes on a large site.
Speed helps
When your server responds quickly and reliably, crawlers can fetch more pages without overloading it. Slow responses and errors make them back off.
Do not over-optimise
For most small and medium sites, the best crawl budget strategy is simply a clean site: no broken links, no endless URLs, a good sitemap and fast pages.
How to use it well
- Fix broken links and redirect chains.
- Block endless parameter URLs in robots.txt or keep them out of links.
- Keep the sitemap clean.
- Make the server fast: slow responses reduce how much gets crawled.
A site audit shows how many pages crawlers would reach from your home page and sitemap, and what wastes their time.
0 comments