Term · Search & content
Crawl budget
Crawl budget describes how many URLs a search engine crawls on a website within a period. It depends on how much crawling the server can handle without slowing down (crawl capacity) and how much the search engine wants to crawl based on the site's popularity and freshness (crawl demand).
[ 01 ]
Explained
When it matters
For most small and medium websites — hundreds or a few thousand pages — crawl budget is rarely a problem; Google says so in its documentation. It becomes important for:
- large online stores with many products and filter combinations;
- sites that generate many URLs automatically;
- sites with slow servers or frequent errors;
- sites that publish many new pages quickly.
What wastes crawl budget
- Endless combinations of filters, sorting and parameters.
- Duplicate URLs: tracking parameters, session IDs, trailing slash variants.
- Redirect chains and broken links.
- Soft 404s — pages returning 200 for missing content.
- Low-value pages: empty categories, thin tag pages.
How to manage it
- Keep filter and parameter URLs out of crawling where they add no value.
- Use canonical tags and consistent internal links.
- Keep sitemaps clean and current. See XML sitemap.
- Fix server errors and speed up responses.
- Return correct status codes for removed pages.
[ 02 ]
FAQ
Frequently asked questions
How do we see crawling activity?
Google Search Console's crawl stats report and server logs show what crawlers request.
Does blocking pages in robots.txt save budget?
It prevents crawling of those URLs, which can help. It does not remove already indexed pages; use noindex or removal for that.