RankEcho

AI crawlers, search and SEO bots

AI crawlers, search and SEO bots, from real logs

Each crawler that reads websites for AI answers, search or SEO tools: who runs it, what it is for, and what it really did on rankecho.io over 30 days of Cloudflare logs. From 17 October to 15 November 2026, these crawlers made 30,984 verified requests to rankecho.io, while 4,505 more used their names without being verified.

Crawlers and tokens40one page each
Verified requests30,98430 days on rankecho.io
Requests faking a name4,505user agent claimed, not verified
Busiest crawlerAmazonbot5,914 verified requests

The crawlers

Crawlers on rankecho.io, 17 October to 15 November 2026
CrawlerOperatorWhat it doesVerified requests, 30 daysSites blocking it, of 13
GPTBotOpenAICollects pages for AI training1343
OAI-SearchBotOpenAIBuilds an AI search index1661
ChatGPT-UserOpenAIFetches a page when a user asks4481
ClaudeBotAnthropicCollects pages for AI training7613
Claude-UserAnthropicFetches a page when a user asks00
Claude-SearchBotAnthropicBuilds an AI search index00
PerplexityBotPerplexityBuilds an AI search index01
Perplexity-UserPerplexityFetches a page when a user asks00
GooglebotGoogleSearch engine crawler1,7510
Google-ExtendedGooglerobots.txt control token, no crawler of its ownNever visits1
BingbotMicrosoftSearch engine crawler1,4231
ApplebotAppleSearch engine crawler1,2061
AmazonbotAmazonCollects pages for AI training5,9142
Meta-ExternalAgentMetaCollects pages for AI training612
CCBotCommon CrawlCollects pages for AI training02
Applebot-ExtendedApplerobots.txt control token, no crawler of its ownNever visits2
DuckAssistBotDuckDuckGoFetches a page when a user asks00
YouBotYou.comBuilds an AI search index0Not checked
MistralAI-UserMistral AIFetches a page when a user asks00
OAI-AdsBotOpenAIChecks ad landing pages425Not checked
Meta-ExternalFetcherMetaFetches a page when a user asks0Not checked
Google-CloudVertexBotGoogleFetches pages when its user runs a check0Not checked
GoogleOtherGoogleGeneral-purpose crawler631
Google-InspectionToolGoogleFetches pages when its user runs a check457Not checked
Googlebot-ImageGoogleSearch engine crawler36Not checked
AI2BotAi2 (Allen Institute for AI)Collects pages for AI training0Not checked
PetalBotHuaweiSearch engine crawler3,186Not checked
DuckDuckBotDuckDuckGoSearch engine crawler24Not checked
YandexBotYandexSearch engine crawler428Not checked
BaiduspiderBaiduSearch engine crawler2,628Not checked
SeznamBotSeznam.czSearch engine crawler306Not checked
AhrefsBotAhrefsBuilds an SEO tool's index3,586Not checked
AhrefsSiteAuditAhrefsFetches pages when its user runs a check4,490Not checked
SemrushBotSemrushBuilds an SEO tool's index883Not checked
DataForSeoBotDataForSEOBuilds an SEO tool's index208Not checked
MJ12botMajesticBuilds an SEO tool's index135Not checked
DotBotMozBuilds an SEO tool's index115Not checked
facebookexternalhitMetaFetches a page to build a link preview918Not checked
ExaSearchBotExaBuilds an AI search index1,172Not checked
TwitterbotXFetches a page to build a link preview60Not checked

Cloudflare verified no request at all from Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, CCBot, DuckAssistBot, YouBot, MistralAI-User, Meta-ExternalFetcher, Google-CloudVertexBot, AI2Bot in these 30 days, though requests using some of those names arrived. Google-Extended and Applebot-Extended never visit: they are robots.txt tokens obeyed by the fetches of Googlebot and Applebot. SEO tool crawlers made 9,417 of the 30,984 verified requests.

Verified requests: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16. Sites blocking it: the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026.

Training, search and user fetches are different jobs

Blocking a training crawler such as GPTBot or ClaudeBot keeps pages out of future model training but leaves you citable in AI answers. Blocking a search crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can drop you from that assistant's answers. User fetchers such as ChatGPT-User read a page because a person asked, and some operators say robots.txt does not bind them. SEO tool crawlers such as AhrefsBot and SemrushBot feed other people's SEO tools and change nothing in search or AI answers, while link preview fetchers such as facebookexternalhit build the card shown when someone shares your link. Decide crawler by crawler.