Why AI can't see your Cloudflare site
If your site sits behind Cloudflare, bot-management rules can block AI crawlers before they ever read a page — and a blocked crawler can never cite you. The fixes are to explicitly allow the AI crawlers you want (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot), make sure your content is in the raw HTML rather than rendered only in the browser, and then verify each crawler gets a real 200 response with full content.
How Cloudflare can make you invisible to AI
Cloudflare's bot-management and security features are designed to stop unwanted automated traffic, and AI crawlers can get caught in that net — challenged, rate-limited, or blocked outright. There is also a managed setting that blocks known AI bots by default on some configurations. The result is silent: your pages look fine to humans while engines receive an error and simply move on to a competitor they can read.
How to check whether you are blocked
Fetch your own page the way a crawler would — a plain request with a crawler user-agent, no browser — and confirm you get a 200 with the full content, not a challenge page or a near-empty shell. Review your Cloudflare bot-management settings and firewall events for blocks on AI user-agents, and check that any AI-bot toggle is set the way you intend.
Which crawlers to allow
Allow the crawlers behind the engines you want to appear in:
- GPTBot and OAI-SearchBot — OpenAI training and ChatGPT search.
- PerplexityBot and Perplexity-User — Perplexity indexing and live fetches.
- ClaudeBot — Anthropic.
- Googlebot — Google Search and AI Overviews; review Google-Extended separately.
The JavaScript-rendering trap
Even with crawlers allowed, content that only appears after client-side JavaScript runs is often invisible to AI fetchers, which frequently read the raw HTML. If your main content is injected by a framework after load, an engine may receive an almost-empty page. Server-render or pre-render the content you want cited so it is present in the initial HTML.
Verify the fix
After adjusting rules and rendering, re-fetch as each crawler and confirm a 200 with full content, then re-run your target prompts over the following days to watch for citations to appear. Because retrieval engines read live, a genuine fix here can show results quickly.
Frequently asked questions
It can, depending on your configuration and bot-management settings. Some setups challenge or block known AI bots unless you explicitly allow them, so it is worth checking rather than assuming.
Fetch the page with a crawler user-agent and no browser; if you get a challenge page, an error, or a near-empty response instead of full content, it is being blocked or the content is not server-rendered.
Allowing specific, legitimate AI crawlers is narrow and reversible; you keep your other bot protections. It is a targeted allowance, not a blanket opening.