RankEcho

Crawling and indexing

What is crawl budget?

Crawl budget is how many URLs a search engine will crawl on a site in a given time. It depends on how fast the server answers (crawl capacity) and how much the engine wants the pages (crawl demand). Google says it matters mainly for very large sites, or large ones whose pages change daily.

How to read it

By Google's own guidance, crawl budget matters for sites with about a million pages, or ten thousand that change daily. A site of a few hundred pages is crawled fully, and its problems lie elsewhere.

What wastes crawling on any site: endless parameter and filter URLs, redirect chains, and old URLs that still get links. Those are worth fixing for clean data even when budget is not the issue.

Example from rankecho.io

A few hundred pages, thousands of requests

rankecho.io has a few hundred indexable pages, so crawl budget is not its problem. In 30 days Googlebot made 1,751 verified requests to 217 different pages, and 182 of them were for addresses that answer with a 4xx, mostly retired pages.

Amazonbot made 5,914 verified requests to 1,776 different addresses, more than any search engine and far more than the sitemap lists.

Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.

Questions people ask

What is crawl budget?

Crawl budget is how many URLs a search engine will crawl on a site in a given time. It depends on how fast the server answers (crawl capacity) and how much the engine wants the pages (crawl demand). Google says it matters mainly for very large sites, or large ones whose pages change daily.

Do small sites need to worry about crawl budget?

Rarely. If new pages are found within days, crawl budget is not what holds them back.

Does blocking AI crawlers save crawl budget for Google?

No. Each crawler has its own budget. Block a crawler only for what it does with your content.

Related terms

Read more on rankecho.io

See it on your own site

RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.

Definition and example checked 2026-11-16.