Release,
AI crawler pages from our own logs
A page for each of 15 AI crawlers and search bots: what it did on rankecho.io in 30 days of Cloudflare logs, and the robots.txt lines to allow or block it.
What changed
15 crawler pages
GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and ten more, each starting with what the crawler is, who runs it and what it is for. See it
Real log numbers
Each page shows the requests Cloudflare verified on rankecho.io from 17 October to 15 November 2026, with a daily chart, the pages the crawler fetched, and how many requests used its name without being verified.
How many sites block it
Counted from the robots.txt files RankEcho read in its own audits, and always shown next to the number of sites in the sample.
Lines to copy
The exact robots.txt lines to allow or block each crawler, and how to check that a visit really came from it. See it
