AI crawler
What is Claude-SearchBot?
Claude-SearchBot crawls pages to improve the quality of Claude's search results. Blocking Claude-SearchBot keeps your pages out of the index Claude searches, which can make them less likely to appear in Claude's answers.
- Operator
- Anthropic
- What it does
- Builds an AI search index
- robots.txt token
Claude-SearchBot- Verified on rankecho.io, 30 days
- 0 requests
What Claude-SearchBot did on rankecho.io
Cloudflare verified no request from Claude-SearchBot to rankecho.io from 17 October to 15 November 2026. In the same days 46 requests used Claude-SearchBot's user agent without Cloudflare verifying it, and 37 of those asked for secret files such as /.env or compose.yml, which no real crawler needs. Cloudflare checks requests against the operator's published addresses, so an unverified request may come from an address Cloudflare does not know yet, or from someone borrowing the name. Treat them as unconfirmed.
Source: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16.
How many sites block Claude-SearchBot
0 of 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block Claude-SearchBot from the whole site. The sample is small and grows with every audit, so read it as a first look rather than a rate for the web.
Blocking Claude-SearchBot keeps your pages out of the index Claude searches, which can make them less likely to appear in Claude's answers. It does not affect ClaudeBot training crawls.
Allow Claude-SearchBot
User-agent: Claude-SearchBot Allow: /
Block Claude-SearchBot
User-agent: Claude-SearchBot Disallow: /
Put the lines in the robots.txt file at the root of your domain. A more specific group for Claude-SearchBot replaces the rules under User-agent: * for that crawler.
How to check a visit is really Claude-SearchBot
Anthropic publishes the IP addresses its crawlers use. Compare the request IP with that list, or let Cloudflare do it: real requests are marked as verified bots.
Questions about Claude-SearchBot
What is Claude-SearchBot?
Claude-SearchBot crawls pages to improve the quality of Claude's search results.
Should I block Claude-SearchBot?
Blocking Claude-SearchBot keeps your pages out of the index Claude searches, which can make them less likely to appear in Claude's answers. It does not affect ClaudeBot training crawls.
How do I know a visit is really from Claude-SearchBot?
Anthropic publishes the IP addresses its crawlers use. Compare the request IP with that list, or let Cloudflare do it: real requests are marked as verified bots.
How often does Claude-SearchBot visit a site?
On rankecho.io, Cloudflare verified no request from Claude-SearchBot from 17 October to 15 November 2026, although 46 requests used its user agent.
How many sites block Claude-SearchBot?
0 of the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block Claude-SearchBot from the whole site. It is a small sample, so read it as a first look, not a web-wide rate.
Related crawlers
Check what your robots.txt allows
The free crawler policy check reads your robots.txt and says which AI crawlers it lets in. A free audit also asks ChatGPT, Perplexity and Gemini whether they cite you, and marks a missing answer as crawler access when a crawler is blocked.
Facts checked 2026-11-16 against Anthropic's documentation. Log numbers from 17 October to 15 November 2026. Read our crawler log study for the method.
