RankEcho

Crawling and indexing

What does crawl-delay do in robots.txt?

Crawl-delay is a robots.txt line asking a crawler to wait a number of seconds between requests. It is not part of the standard: Googlebot ignores it, while some crawlers such as Bingbot and SemrushBot honour it, Semrush up to 10 seconds. Use it only when a crawler truly strains your server.

How to read it

A delay slows every page a crawler fetches. On a large site that can mean pages are refreshed in its index weeks later, so the cost is freshness.

Set it in the group for the crawler you mean, not under User-agent: *, or you slow the crawlers you want most. For Google, crawl rate follows how fast and healthy your server answers.

Example from rankecho.io

No delay needed at 29 requests a day

SemrushBot made 883 verified requests to rankecho.io in 30 days, about 29 a day, and read robots.txt 345 times. Spread over a day, that is no load at all.

rankecho.io's robots.txt sets no crawl-delay for any crawler. A delay earns its place when a crawler sends bursts your server cannot answer, and the logs show the bursts first.

Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.

Questions people ask

What does crawl-delay do in robots.txt?

Crawl-delay is a robots.txt line asking a crawler to wait a number of seconds between requests. It is not part of the standard: Googlebot ignores it, while some crawlers such as Bingbot and SemrushBot honour it, Semrush up to 10 seconds. Use it only when a crawler truly strains your server.

Does Google obey crawl-delay?

No. Googlebot ignores it. Google adjusts its crawl rate to how your server responds.

What crawl-delay value should I use?

Usually none. If one crawler strains your server, start with a few seconds in that crawler's own group.

Related terms

Read more on rankecho.io

See it on your own site

RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.

Definition and example checked 2026-11-16.