RankEcho

Crawling and indexing

What is robots.txt?

robots.txt is a plain text file at the root of a site, such as example.com/robots.txt, that tells crawlers which paths they may fetch. Groups name a crawler by its token and list Allow and Disallow rules. Well-behaved crawlers obey it; it does not hide pages or stop indexing on its own.

How to read it

robots.txt controls crawling, not indexing. A blocked page can still be indexed from links to it, without its content. To keep a page out of results, let it be crawled and mark it noindex.

Crawlers fetch robots.txt again and again, because they may only cache it for a short time. A file that fails to load, with a 5xx error, can make a crawler stop fetching the whole site until it loads again.

Example from rankecho.io

Who read rankecho.io's robots.txt

In 30 days, verified Googlebot requests read rankecho.io's robots.txt 379 times, SemrushBot 345 times, AhrefsBot 161 times and ClaudeBot 157 times.

GPTBot, in the same days, made 134 verified requests and read robots.txt not once from a verified address. Reading frequency varies widely between crawlers; what matters is that the file is always there and always loads.

Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.

Questions people ask

What is robots.txt?

robots.txt is a plain text file at the root of a site, such as example.com/robots.txt, that tells crawlers which paths they may fetch. Groups name a crawler by its token and list Allow and Disallow rules. Well-behaved crawlers obey it; it does not hide pages or stop indexing on its own.

Does robots.txt stop a page being indexed?

No. It stops crawling. A blocked page can still appear in results from links to it. Use noindex on a crawlable page to keep it out.

Where must robots.txt be?

At the root of each host, for example https://www.example.com/robots.txt. A subdomain needs its own file.

Related terms

Read more on rankecho.io

See it on your own site

RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.

Definition and example checked 2026-11-16.