Why AI can't see your Cloudflare site
Cloudflare can serve a crawler a block, challenge, rate limit, or different body even when robots.txt permits the URL. Diagnose the layers separately: served robots policy, AI Crawl Control, WAF rule order, verified bot identity, edge status and body, and origin logs. An allowed rule or spoofed user-agent 200 does not prove genuine reachability, indexing, or citation.
Which Cloudflare layer controls the request?
Robots.txt and Cloudflare managed robots communicate preferences. AI Crawl Control and WAF rules enforce requests at the edge. Rate limiting, custom rules, bot products, Workers, cache behavior, and the origin can each change the outcome. Cloudflare documents that an allowance in AI Crawl Control can still be blocked by another WAF rule.
Start with the exact edge event for the target URL and time. Do not infer the cause from a generic Cloudflare default, a robots file alone, or the fact that a browser request succeeds.
How do I diagnose a Cloudflare crawler failure?
Use a verification ladder and save the evidence at each step. The crawler-policy checker can parse the current robots.txt file, but it cannot authenticate a bot or inspect your Cloudflare account, WAF, cache, logs, indexing state, or citation outcome.
- 1. Fetch the served /robots.txt and evaluate the exact bot and target path
- 2. Review AI Crawl Control preferences by Search, Agent, and Training behavior
- 3. Inspect Security Events and WAF rule order for the request
- 4. Verify the genuine crawler with provider IP ranges, reverse DNS, or a verified-bot signal
- 5. Compare edge and origin status, headers, canonical, and useful HTML body
- 6. Confirm the crawler reached the origin or document why it was intentionally stopped
- 7. Check provider indexing tools and AI outcomes separately
Why is a bot-shaped curl request not enough?
Anyone can send User-Agent: OAI-SearchBot or Googlebot. That request is useful for testing how a header and URL move through your stack, but it does not prove what the authentic crawler received. Match the source to official provider ranges or verification guidance and reconcile it with Cloudflare and origin logs.
A 200 status also needs a body check. The response could be a managed challenge, consent wall, login page, cached error, empty application shell, wrong canonical, or noindex response. Record status, content type, final URL, headers, and representative body text.
Which crawler policies should I configure?
Separate provider roles instead of applying one AI-bot switch to every use:
- Search discovery: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot
- Training choices: GPTBot and ClaudeBot; assess other providers from their current documentation
- User actions: ChatGPT-User and Perplexity-User have robots caveats; Claude-User is documented as honoring robots
- Control token: Google-Extended is separate from Google Search and has no distinct HTTP user agent
How do I make a narrow, reversible Cloudflare change?
Use the smallest rule that resolves the verified event: the intended bot identity, public path, action, and priority. Preserve authentication and private-path controls. Log the old and new configuration, deploy in a limited window, test representative public and private URLs, and keep a rollback ready.
Monitor edge events, origin load, error rates, and unexpected paths after the change. If the genuine bot never reappears, that absence is not proof of a block; crawl schedules and demand are outside your control.
What if the page body depends on JavaScript?
Crawler identity and rendering are different questions. Inspect the initial HTML from the verified request and determine whether the useful answer is present. If only a client-side shell is returned, server rendering or prerendering may improve access for clients that do not execute the application, but it does not guarantee indexing or citation.
When is the Cloudflare fix proven?
The technical fix is evidenced when the intended, verified request follows the expected rule and the edge and origin return the expected public content without exposing private routes. Search indexing, AI impressions, and citations are later outcomes with other causes. Record them separately and avoid assigning causation from sequence alone.
Sources reviewed
Material crawler-role and control claims below were checked against primary provider documentation and the robots standard. Access settings affect eligibility and reachability; they do not guarantee indexing, ranking, an AI impression, or a citation.
13 claim-level source records
| Claim reviewed | Official source | Review record |
|---|---|---|
| OpenAI documents OAI-SearchBot for ChatGPT search, GPTBot for potential model training, and ChatGPT-User for user-triggered actions; the controls are independent and robots.txt rules may not apply to ChatGPT-User. | OpenAI crawler documentation | Checked 2026-09-01 · Current OAI-SearchBot, GPTBot, and ChatGPT-User documentation · Primary-source documentation review; no claim that a permitted bot will index, rank, or cite a page. · Confidence: High |
| Anthropic documents ClaudeBot for potential model training, Claude-SearchBot for search indexing, and Claude-User for user-directed retrieval; Anthropic says all three honor robots.txt. | Anthropic crawler documentation | Checked 2026-09-01 · Crawler taxonomy updated 2026-04-07 · Primary-source documentation review; crawler permission is treated as access policy, not citation eligibility. · Confidence: High |
| Perplexity documents PerplexityBot for search discovery and Perplexity-User for user-triggered fetching; Perplexity-User generally ignores robots.txt and is not a crawl or training bot. | Perplexity crawler documentation | Checked 2026-09-01 · Current PerplexityBot and Perplexity-User documentation · Primary-source documentation review; Perplexity recommends checking both the user agent and published IP ranges. · Confidence: High |
| Microsoft lists Bingbot as Bing's main web crawler. Permitting Bingbot supports Bing discovery but does not guarantee indexing, an impression, or a citation. | Bing crawler documentation | Checked 2026-09-01 · Current Bing crawler documentation · Primary-source documentation review; Bingbot access and index eligibility do not guarantee a Copilot citation or impression. · Confidence: High |
| Microsoft's public-website guidance for Copilot Studio says website knowledge sources rely on Bing discovery and should be accessible to Bingbot. | Microsoft Copilot Studio public-website guidance | Checked 2026-09-01 · Current public-website knowledge-source guidance · Primary-source documentation review; Bing discovery is one dependency and does not guarantee a Copilot answer, impression, or citation. · Confidence: High |
| Google says normal Search indexing and snippet controls govern eligibility for AI Overviews and AI Mode; there are no additional technical requirements or special AI schema files. | Google Search AI features documentation | Checked 2026-09-01 · AI features documentation updated 2025-12-10 · Primary-source documentation review; eligibility does not guarantee selection or presentation in an AI feature. · Confidence: High |
| Google documents Google-Extended as a robots.txt control for some Gemini Apps, Vertex AI grounding, and future model training uses; it has no separate HTTP user agent and does not affect Google Search inclusion or ranking. | Google common crawler documentation | Checked 2026-09-01 · Google-Extended documentation updated 2026-07-14 · Primary-source documentation review; Google-Extended is not represented as the control for Google Search AI features. · Confidence: High |
| Cloudflare AI Crawl Control exposes per-crawler activity and allow or block controls at the edge. | Cloudflare AI Crawl Control documentation | Checked 2026-09-01 · Current AI Crawl Control documentation · Primary-source documentation review; account configuration and activity must be checked rather than inferred from a generic default. · Confidence: High |
| Cloudflare documents separate Search, Agent, and Training policy presets. Its updated defaults for new domains are scheduled for September 15, 2026 and were not yet effective on this September 1 review date. | Cloudflare AI bot policy documentation | Checked 2026-09-01 · Policy documentation updated 2026-07-01; future-default boundary checked 2026-09-01 · Primary-source documentation review; current zone settings must be inspected and future defaults are not represented as active. · Confidence: High |
| Cloudflare documents managed robots.txt as preference signaling whose directives remain voluntary for crawlers; enforcement requires an edge control rather than the file alone. | Cloudflare managed robots.txt documentation | Checked 2026-09-01 · Managed robots.txt documentation updated 2026-08-03 · Primary-source documentation review; the served file must be checked for the specific zone and rollout state. · Confidence: High |
| Cloudflare documents that a crawler allowed by AI Crawl Control can still be blocked by a custom WAF rule, so rule order and the matching security event must be reviewed. | Cloudflare AI Crawl Control with WAF | Checked 2026-09-01 · Current AI Crawl Control and WAF configuration guidance · Primary-source documentation review; the result depends on the zone's actual rules and request evidence. · Confidence: High |
| A crawler user-agent string can be spoofed; Google documents reverse-DNS and published-IP methods for verifying genuine Google crawlers. | Google crawler verification documentation | Checked 2026-09-01 · Current verified-crawler guidance · Primary-source documentation review; a curl request that only changes User-Agent is a path test, not bot-identity verification. · Confidence: High |
| RFC 9309 defines robots.txt matching, including merging multiple groups for the same user agent and using the most specific matching rule. | Robots Exclusion Protocol, RFC 9309 | Checked 2026-09-01 · IETF standards-track RFC published September 2022 · Primary-standard review; simple policy checkers may not implement every URL-level matching case. · Confidence: High |
Frequently asked questions
Do not assume a universal default. Cloudflare behavior depends on zone settings, product configuration, rule order, and rollout date. Inspect AI Crawl Control, WAF events, the served robots file, and the exact request.
Verify the requester using provider IP or reverse-DNS guidance, then reconcile the exact event across Cloudflare and origin logs. A user-agent string alone is spoofable.
No. Inspect the final URL, headers, canonical, directives, content type, and useful HTML body. A 200 can contain a challenge, login page, empty shell, or other unusable response.
No guarantee follows from access. It may affect eligibility for systems that use that crawler, while indexing, retrieval, source selection, demand, and answer variation remain separate.