How to allow AI crawlers in robots.txt
To allow AI search crawlers, permit public paths for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, and Googlebot while repeating non-public crawl exclusions in their named groups. Secure those routes independently, decide training access separately, then verify CDN rules and genuine bot identity. Robots.txt expresses crawl policy; it does not prove reachability, indexing, impressions, or citations.
Which AI crawlers should I allow?
Choose by documented role and by your own publishing policy. Search crawlers affect whether provider search systems can discover public pages. Training crawlers are a separate rights decision. User-action fetchers act after a person requests a page, and their robots behavior is provider-specific. A control token such as Google-Extended is not necessarily an HTTP crawler.
| Token | Documented role | Robots behavior | Policy decision |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search discovery | Honors robots.txt | Allow public pages for ChatGPT search eligibility |
| GPTBot | Potential OpenAI model training | Honors robots.txt | Separate training-rights choice |
| ChatGPT-User | User-triggered OpenAI fetch | Rules may not apply | Do not infer reachability from robots.txt |
| Claude-SearchBot | Claude search indexing | Anthropic says it honors robots.txt | Allow public pages if Claude discovery matters |
| ClaudeBot | Potential Anthropic model training | Anthropic says it honors robots.txt | Separate training-rights choice |
| Claude-User | User-directed Claude retrieval | Anthropic says it honors robots.txt | Separate from Claude search indexing |
| PerplexityBot | Perplexity search discovery | Honors robots.txt | Allow public pages if Perplexity discovery matters |
| Perplexity-User | User-triggered Perplexity fetch | Generally ignores robots.txt | Do not treat as a robots-controlled crawler |
| Googlebot | Google Search, AI Overviews, and AI Mode | Honors Google Search robots controls | Use normal Search indexing and snippet controls |
| Google-Extended | Gemini/Vertex use-control token | Robots control; no separate HTTP user agent | Separate from Google Search eligibility |
| Bingbot | Bing Search discovery and Bing-grounded surfaces | Honors robots.txt | Allow public pages if Bing discovery matters |
What should the robots.txt look like?
A safe example permits public content for search crawlers while repeating private-path exclusions inside the exact group. That repetition matters: under the Robots Exclusion Protocol, exact user-agent groups are selected instead of the wildcard group, so a private rule placed only under User-agent: * may not protect the same path from a named bot.
Adapt the private paths to your application. Robots.txt is public and is not access control; authentication and authorization must protect sensitive routes even when the file lists them.
- User-agent: OAI-SearchBot
- User-agent: Claude-SearchBot
- User-agent: PerplexityBot
- User-agent: Bingbot
- User-agent: Googlebot
- Allow: /
- Disallow: /api/
- Disallow: /dashboard/
- Disallow: /login
- Sitemap: https://yourdomain.com/sitemap.xml
How do I allow all compliant crawlers while keeping sensitive routes secured?
Use a wildcard group for public content, repeat the same crawl exclusions, and enforce sensitive routes in the application. Robots.txt is public, so every listed path is discoverable; exclusions communicate a preference and do not hide or secure the route. Do not publish a blanket Allow: / and assume it is harmless: compliant bots may then request any path that is not protected elsewhere.
If the file also contains named groups, audit every exact group for the private exclusions. RFC 9309 merges multiple groups that match the same user agent and applies the most specific path rule; a simple root-level checker may not evaluate every wildcard, end anchor, encoded path, or URL-specific case.
- User-agent: *
- Allow: /
- Disallow: /api/
- Disallow: /dashboard/
- Disallow: /login
Is Cloudflare blocking a crawler that robots.txt allows?
It can. Managed robots.txt records a preference, while AI Crawl Control, WAF custom rules, rate limits, and other edge controls enforce requests before the origin responds. Cloudflare also notes that an AI Crawl Control allowance can still lose to another WAF rule, so inspect the rule order and the actual edge event rather than relying on one toggle.
Avoid assumptions about plan-wide or account-wide defaults. Review the current Search, Agent, and Training preferences, the served robots.txt file, WAF events, and origin logs for your own zone.
How do I verify genuine crawler access?
Verify in layers. First parse the rule for the exact token and target URL. Then inspect CDN and WAF events. Confirm genuine crawler identity with the provider's published IP ranges, reverse DNS, or verified-bot signal where documented. Finally, check the status, headers, and response body in edge and origin logs.
A curl request with a bot-shaped User-Agent only tests how your stack handled that supplied header and path; the header can be spoofed. A 200 can still contain a challenge, login page, empty shell, noindex directive, or wrong canonical. None of these checks proves indexing or citation, so use provider webmaster tools and repeated outcome tests as separate steps.
- 1. Evaluate the exact robots group and target path
- 2. Inspect CDN, AI Crawl Control, and WAF events
- 3. Verify bot identity against official provider guidance
- 4. Confirm status, headers, canonical, and useful HTML body
- 5. Reconcile edge events with origin access logs
- 6. Check Google Search Console or Bing Webmaster Tools where applicable
- 7. Re-test AI visibility as an outcome, not as proof of causation
Does crawler access guarantee AI impressions or citations?
No. Access is one possible prerequisite for systems that discover pages through that crawler, but providers still decide whether to index, retrieve, select, summarize, or cite the page. Google specifically says AI Overviews and AI Mode use normal Search eligibility and snippet controls, with no special AI schema or text-file requirement, and eligibility does not guarantee appearance.
If impressions or citations fall after access is verified, investigate indexing, canonical and snippet controls, page usefulness, source selection, demand, and answer variability. Do not attribute the decline to robots.txt from timing alone.
How often should crawler policy be reviewed?
Review after a CDN, WAF, hosting, or robots deployment and on a recurring schedule. Provider names and documented roles change, but there is no universal interval or citation-update deadline. Record the provider source, review date, rule change, and verification evidence so a later drop can be investigated against a real baseline.
Sources reviewed
Material crawler-role and control claims below were checked against primary provider documentation and the robots standard. Access settings affect eligibility and reachability; they do not guarantee indexing, ranking, an AI impression, or a citation.
13 claim-level source records
| Claim reviewed | Official source | Review record |
|---|---|---|
| OpenAI documents OAI-SearchBot for ChatGPT search, GPTBot for potential model training, and ChatGPT-User for user-triggered actions; the controls are independent and robots.txt rules may not apply to ChatGPT-User. | OpenAI crawler documentation | Checked 2026-09-01 · Current OAI-SearchBot, GPTBot, and ChatGPT-User documentation · Primary-source documentation review; no claim that a permitted bot will index, rank, or cite a page. · Confidence: High |
| Anthropic documents ClaudeBot for potential model training, Claude-SearchBot for search indexing, and Claude-User for user-directed retrieval; Anthropic says all three honor robots.txt. | Anthropic crawler documentation | Checked 2026-09-01 · Crawler taxonomy updated 2026-04-07 · Primary-source documentation review; crawler permission is treated as access policy, not citation eligibility. · Confidence: High |
| Perplexity documents PerplexityBot for search discovery and Perplexity-User for user-triggered fetching; Perplexity-User generally ignores robots.txt and is not a crawl or training bot. | Perplexity crawler documentation | Checked 2026-09-01 · Current PerplexityBot and Perplexity-User documentation · Primary-source documentation review; Perplexity recommends checking both the user agent and published IP ranges. · Confidence: High |
| Microsoft lists Bingbot as Bing's main web crawler. Permitting Bingbot supports Bing discovery but does not guarantee indexing, an impression, or a citation. | Bing crawler documentation | Checked 2026-09-01 · Current Bing crawler documentation · Primary-source documentation review; Bingbot access and index eligibility do not guarantee a Copilot citation or impression. · Confidence: High |
| Microsoft's public-website guidance for Copilot Studio says website knowledge sources rely on Bing discovery and should be accessible to Bingbot. | Microsoft Copilot Studio public-website guidance | Checked 2026-09-01 · Current public-website knowledge-source guidance · Primary-source documentation review; Bing discovery is one dependency and does not guarantee a Copilot answer, impression, or citation. · Confidence: High |
| Google says normal Search indexing and snippet controls govern eligibility for AI Overviews and AI Mode; there are no additional technical requirements or special AI schema files. | Google Search AI features documentation | Checked 2026-09-01 · AI features documentation updated 2025-12-10 · Primary-source documentation review; eligibility does not guarantee selection or presentation in an AI feature. · Confidence: High |
| Google documents Google-Extended as a robots.txt control for some Gemini Apps, Vertex AI grounding, and future model training uses; it has no separate HTTP user agent and does not affect Google Search inclusion or ranking. | Google common crawler documentation | Checked 2026-09-01 · Google-Extended documentation updated 2026-07-14 · Primary-source documentation review; Google-Extended is not represented as the control for Google Search AI features. · Confidence: High |
| Cloudflare AI Crawl Control exposes per-crawler activity and allow or block controls at the edge. | Cloudflare AI Crawl Control documentation | Checked 2026-09-01 · Current AI Crawl Control documentation · Primary-source documentation review; account configuration and activity must be checked rather than inferred from a generic default. · Confidence: High |
| Cloudflare documents separate Search, Agent, and Training policy presets. Its updated defaults for new domains are scheduled for September 15, 2026 and were not yet effective on this September 1 review date. | Cloudflare AI bot policy documentation | Checked 2026-09-01 · Policy documentation updated 2026-07-01; future-default boundary checked 2026-09-01 · Primary-source documentation review; current zone settings must be inspected and future defaults are not represented as active. · Confidence: High |
| Cloudflare documents managed robots.txt as preference signaling whose directives remain voluntary for crawlers; enforcement requires an edge control rather than the file alone. | Cloudflare managed robots.txt documentation | Checked 2026-09-01 · Managed robots.txt documentation updated 2026-08-03 · Primary-source documentation review; the served file must be checked for the specific zone and rollout state. · Confidence: High |
| Cloudflare documents that a crawler allowed by AI Crawl Control can still be blocked by a custom WAF rule, so rule order and the matching security event must be reviewed. | Cloudflare AI Crawl Control with WAF | Checked 2026-09-01 · Current AI Crawl Control and WAF configuration guidance · Primary-source documentation review; the result depends on the zone's actual rules and request evidence. · Confidence: High |
| A crawler user-agent string can be spoofed; Google documents reverse-DNS and published-IP methods for verifying genuine Google crawlers. | Google crawler verification documentation | Checked 2026-09-01 · Current verified-crawler guidance · Primary-source documentation review; a curl request that only changes User-Agent is a path test, not bot-identity verification. · Confidence: High |
| RFC 9309 defines robots.txt matching, including merging multiple groups for the same user agent and using the most specific matching rule. | Robots Exclusion Protocol, RFC 9309 | Checked 2026-09-01 · IETF standards-track RFC published September 2022 · Primary-standard review; simple policy checkers may not implement every URL-level matching case. · Confidence: High |
Frequently asked questions
Use a User-agent: * group that allows public content and repeats private-path exclusions such as /api/, /dashboard/, and /login. Named groups must also preserve those exclusions. Robots.txt is not security; sensitive routes still require authentication and authorization.
GPTBot is documented for crawling that may support model training. OAI-SearchBot is documented for ChatGPT search discovery. Their controls are independent, so a training opt-out does not by itself describe ChatGPT search eligibility.
OpenAI says robots.txt rules may not apply to ChatGPT-User because it handles user-triggered requests rather than automatic search crawling. OAI-SearchBot is the documented control for ChatGPT search eligibility.
No. Google says normal Search controls through Googlebot govern AI Overviews and AI Mode. Google-Extended controls certain Gemini, Vertex AI, and future-training uses and does not affect Google Search inclusion or ranking.
No. It records policy only. CDN or WAF enforcement, bot identity, target-path matching, response content, indexing controls, and provider selection are separate. A spoofable user-agent request cannot prove what a genuine crawler received.