Why AI can't see your Cloudflare site — and how to fix it
If AI never cites your site, the cause is often that AI crawlers are blocked — either by a Disallow rule in robots.txt or by CDN bot-management settings. If GPTBot, ClaudeBot, PerplexityBot, or Google-Extended can't fetch your pages, no amount of content will get you cited.
How do I check if AI crawlers are blocked?
Read your robots.txt for Disallow rules affecting AI user-agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended), and check whether your CDN or bot-management is challenging or rejecting those agents.
How do I fix it?
Explicitly allow the AI crawlers you want to be cited by, ensure your important content is server-side rendered in raw HTML, and confirm nothing important sits behind a login or JavaScript-only render.
Frequently asked questions
Bot-management and AI-scraping controls can restrict them depending on your configuration, so it's worth verifying rather than assuming access.
No. Allowing answer-engine crawlers is separate from your Google indexing and is what makes AI citation possible.