RankEcho

AI crawler

What is ClaudeBot?

ClaudeBot collects public web pages that may be used to train Anthropic's Claude models. Blocking ClaudeBot opts your future pages out of Anthropic's training crawls.

Operator
Anthropic
What it does
Collects pages for AI training
robots.txt token
ClaudeBot
Verified on rankecho.io, 30 days
761 requests

What ClaudeBot did on rankecho.io

From 17 October to 15 November 2026, Cloudflare verified 761 requests from ClaudeBot, on 16 of the 30 days. It fetched 165 different pages, read robots.txt 157 times, sitemaps 199 times and llms.txt 10 times. 4 of its requests got an error status (4xx or 5xx).

Verified requests76116 of 30 days
Different pages fetched165378 page requests
robots.txt reads157sitemaps 199, llms.txt 10
Requests faking its name120same user agent, not verified
  • Verified requests
Verified ClaudeBot requests per day to rankecho.io, 17 October to 15 November 2026
DateVerified requests
Oct 170
Oct 180
Oct 190
Oct 2040
Oct 21240
Oct 2260
Oct 2310
Oct 2448
Oct 2543
Oct 2649
Oct 2754
Oct 2827
Oct 290
Oct 3040
Oct 3124
Nov 131
Nov 250
Nov 325
Nov 49
Nov 511
Nov 60
Nov 70
Nov 80
Nov 90
Nov 100
Nov 110
Nov 120
Nov 130
Nov 140
Nov 150
Pages ClaudeBot fetched most
PathRequests
/30
/login14
/resources/find10
/compare/ai-citation-tracker10
/benchmark/uclahealth.org10

Also in those days, 120 requests used ClaudeBot's user agent without Cloudflare verifying it, and 88 of those asked for secret files such as /.env or compose.yml, which no real crawler needs.

Source: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16. Cloudflare sampled 350 of the verified requests, so those counts are estimates.

How many sites block ClaudeBot

3 of 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block ClaudeBot from the whole site. The sample is small and grows with every audit, so read it as a first look rather than a rate for the web.

Blocking ClaudeBot opts your future pages out of Anthropic's training crawls. Claude's live answers use Claude-SearchBot and Claude-User, which have their own tokens.

Allow ClaudeBot

User-agent: ClaudeBot
Allow: /

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Put the lines in the robots.txt file at the root of your domain. A more specific group for ClaudeBot replaces the rules under User-agent: * for that crawler.

How to check a visit is really ClaudeBot

Anthropic publishes the IP addresses its crawlers use. Compare the request IP with that list, or let Cloudflare do it: real requests are marked as verified bots.

Source: Anthropic: Does Anthropic crawl data from the web?.

Questions about ClaudeBot

What is ClaudeBot?

ClaudeBot collects public web pages that may be used to train Anthropic's Claude models.

Should I block ClaudeBot?

Blocking ClaudeBot opts your future pages out of Anthropic's training crawls. Claude's live answers use Claude-SearchBot and Claude-User, which have their own tokens.

How do I know a visit is really from ClaudeBot?

Anthropic publishes the IP addresses its crawlers use. Compare the request IP with that list, or let Cloudflare do it: real requests are marked as verified bots.

How often does ClaudeBot visit a site?

On rankecho.io, ClaudeBot made 761 Cloudflare-verified requests from 17 October to 15 November 2026, on 16 of the 30 days. Your site will differ with its size and how often it changes.

How many sites block ClaudeBot?

3 of the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block ClaudeBot from the whole site. It is a small sample, so read it as a first look, not a web-wide rate.

Related crawlers

Check what your robots.txt allows

The free crawler policy check reads your robots.txt and says which AI crawlers it lets in. A free audit also asks ChatGPT, Perplexity and Gemini whether they cite you, and marks a missing answer as crawler access when a crawler is blocked.

Facts checked 2026-11-16 against Anthropic's documentation. Log numbers from 17 October to 15 November 2026. Read our crawler log study for the method.