AI crawlers, search and SEO bots
AI crawlers, search and SEO bots, from real logs
Each crawler that reads websites for AI answers, search or SEO tools: who runs it, what it is for, and what it really did on rankecho.io over 30 days of Cloudflare logs. From 17 October to 15 November 2026, these crawlers made 30,984 verified requests to rankecho.io, while 4,505 more used their names without being verified.
The crawlers
| Crawler | Operator | What it does | Verified requests, 30 days | Sites blocking it, of 13 |
|---|---|---|---|---|
| GPTBot | OpenAI | Collects pages for AI training | 134 | 3 |
| OAI-SearchBot | OpenAI | Builds an AI search index | 166 | 1 |
| ChatGPT-User | OpenAI | Fetches a page when a user asks | 448 | 1 |
| ClaudeBot | Anthropic | Collects pages for AI training | 761 | 3 |
| Claude-User | Anthropic | Fetches a page when a user asks | 0 | 0 |
| Claude-SearchBot | Anthropic | Builds an AI search index | 0 | 0 |
| PerplexityBot | Perplexity | Builds an AI search index | 0 | 1 |
| Perplexity-User | Perplexity | Fetches a page when a user asks | 0 | 0 |
| Googlebot | Search engine crawler | 1,751 | 0 | |
| Google-Extended | robots.txt control token, no crawler of its own | Never visits | 1 | |
| Bingbot | Microsoft | Search engine crawler | 1,423 | 1 |
| Applebot | Apple | Search engine crawler | 1,206 | 1 |
| Amazonbot | Amazon | Collects pages for AI training | 5,914 | 2 |
| Meta-ExternalAgent | Meta | Collects pages for AI training | 61 | 2 |
| CCBot | Common Crawl | Collects pages for AI training | 0 | 2 |
| Applebot-Extended | Apple | robots.txt control token, no crawler of its own | Never visits | 2 |
| DuckAssistBot | DuckDuckGo | Fetches a page when a user asks | 0 | 0 |
| YouBot | You.com | Builds an AI search index | 0 | Not checked |
| MistralAI-User | Mistral AI | Fetches a page when a user asks | 0 | 0 |
| OAI-AdsBot | OpenAI | Checks ad landing pages | 425 | Not checked |
| Meta-ExternalFetcher | Meta | Fetches a page when a user asks | 0 | Not checked |
| Google-CloudVertexBot | Fetches pages when its user runs a check | 0 | Not checked | |
| GoogleOther | General-purpose crawler | 63 | 1 | |
| Google-InspectionTool | Fetches pages when its user runs a check | 457 | Not checked | |
| Googlebot-Image | Search engine crawler | 36 | Not checked | |
| AI2Bot | Ai2 (Allen Institute for AI) | Collects pages for AI training | 0 | Not checked |
| PetalBot | Huawei | Search engine crawler | 3,186 | Not checked |
| DuckDuckBot | DuckDuckGo | Search engine crawler | 24 | Not checked |
| YandexBot | Yandex | Search engine crawler | 428 | Not checked |
| Baiduspider | Baidu | Search engine crawler | 2,628 | Not checked |
| SeznamBot | Seznam.cz | Search engine crawler | 306 | Not checked |
| AhrefsBot | Ahrefs | Builds an SEO tool's index | 3,586 | Not checked |
| AhrefsSiteAudit | Ahrefs | Fetches pages when its user runs a check | 4,490 | Not checked |
| SemrushBot | Semrush | Builds an SEO tool's index | 883 | Not checked |
| DataForSeoBot | DataForSEO | Builds an SEO tool's index | 208 | Not checked |
| MJ12bot | Majestic | Builds an SEO tool's index | 135 | Not checked |
| DotBot | Moz | Builds an SEO tool's index | 115 | Not checked |
| facebookexternalhit | Meta | Fetches a page to build a link preview | 918 | Not checked |
| ExaSearchBot | Exa | Builds an AI search index | 1,172 | Not checked |
| Twitterbot | X | Fetches a page to build a link preview | 60 | Not checked |
Cloudflare verified no request at all from Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, CCBot, DuckAssistBot, YouBot, MistralAI-User, Meta-ExternalFetcher, Google-CloudVertexBot, AI2Bot in these 30 days, though requests using some of those names arrived. Google-Extended and Applebot-Extended never visit: they are robots.txt tokens obeyed by the fetches of Googlebot and Applebot. SEO tool crawlers made 9,417 of the 30,984 verified requests.
Verified requests: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16. Sites blocking it: the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026.
Training, search and user fetches are different jobs
Blocking a training crawler such as GPTBot or ClaudeBot keeps pages out of future model training but leaves you citable in AI answers. Blocking a search crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can drop you from that assistant's answers. User fetchers such as ChatGPT-User read a page because a person asked, and some operators say robots.txt does not bind them. SEO tool crawlers such as AhrefsBot and SemrushBot feed other people's SEO tools and change nothing in search or AI answers, while link preview fetchers such as facebookexternalhit build the card shown when someone shares your link. Decide crawler by crawler.
