Which AI crawlers do websites block? What 2,000 robots.txt files show
Few, and mostly the ones that collect training data. We read robots.txt on the 1,000 sites AI engines cited most in RankEcho audits, mostly marketing, SEO and software sites, and on 1,000 small-business sites in November 2026. 10.5% of the cited sites and 5.4% of the small-business sites block any AI bot. Of those, about three in four block only training crawlers such as GPTBot, CCBot and Bytespider while still letting AI search crawlers in: 33 of the 43 cited sites that block GPTBot still allow OpenAI's search crawler. Googlebot and Bingbot are almost never blocked.
Key findings
Two cohorts of 1,000 sites each, read on 16 and 17 November 2026: the hosts AI engines cited most often in RankEcho audit answers (44,371 citations, mostly in answers about marketing and SEO software), and small-business homepages from public business listings. Shares are of the sites whose robots.txt answered: 988 and 990.
- Blocking is rare. 104 of the cited sites (10.5%) and 53 small-business sites (5.4%) block at least one of the 22 bots we checked.
- Sites block training, not search. 79 of the 104 cited sites that block anything (76%) block only training crawlers. Only 21 (2.1%) block an AI search crawler.
- The split is on purpose. 33 of the 43 cited sites that block GPTBot allow OAI-SearchBot, and 30 of 31 that block Google-Extended allow Googlebot. Among small businesses it is 42 of 46.
- The most-blocked bot differs. Among the cited sites it is ByteDance's Bytespider (80 sites); among small businesses it is GPTBot (46).
- Search engines are left alone. Googlebot is blocked by 1 cited site and 2 small-business sites, Bingbot by 2 each.
- Many small businesses have no rules at all. 109 small-business sites (11%) had no robots.txt, which lets every bot in, against 46 of the cited sites.
What share of sites block each kind of bot
We sorted the 22 robots.txt tokens by what their operators say they are for. Training crawlers collect pages for model training. AI search crawlers build the indexes that ChatGPT search, Claude, Perplexity and DuckAssist answer from. User-fetch agents load a page when a person asks an assistant about it. A site that blocks the last two is asking to be left out of AI answers, not only out of training data.
| Blocks | Cited sites | Small-business sites |
|---|---|---|
| Any of the 22 bots | 104 (10.5%) | 53 (5.4%) |
| A training crawler | 102 (10.3%) | 50 (5.1%) |
| An AI search crawler | 21 (2.1%) | 11 (1.1%) |
| A user-fetch agent | 21 (2.1%) | 8 (0.8%) |
| Googlebot or Bingbot | 2 (0.2%) | 2 (0.2%) |
Which bots are blocked most
Each training crawler is blocked by about 3% to 8% of the cited sites, with Bytespider and Common Crawl's CCBot at the top. The AI search crawlers and user-fetch agents sit between 1% and 1.5%. Amazonbot, which Amazon says may also be used to train AI models, is blocked about as often as the training crawlers.
Sites that block anything tend to block many bots at once: 5.6 on average among the cited sites and 9.3 among small businesses, which suggests ready-made block lists copied from plugins or templates.
| Bot | Operator | Kind | Cited sites | Small-business sites |
|---|---|---|---|---|
| Bytespider | ByteDance | Training crawler | 80 | 44 |
| CCBot | Common Crawl | Training crawler | 55 | 43 |
| GPTBot | OpenAI | Training crawler | 43 | 46 |
| Meta-ExternalAgent | Meta | Training crawler | 41 | 42 |
| Applebot-Extended | Apple | Training crawler | 38 | 41 |
| anthropic-ai | Anthropic | Training crawler | 37 | 38 |
| ClaudeBot | Anthropic | Training crawler | 36 | 41 |
| cohere-ai | Cohere | Training crawler | 34 | 37 |
| Google-Extended | Training crawler | 31 | 37 | |
| GoogleOther | Training crawler | 28 | 32 | |
| Amazonbot | Amazon | Mixed use | 47 | 42 |
| PerplexityBot | Perplexity | AI search crawler | 15 | 9 |
| DuckAssistBot | DuckDuckGo | AI search crawler | 12 | 7 |
| OAI-SearchBot | OpenAI | AI search crawler | 11 | 4 |
| Claude-SearchBot | Anthropic | AI search crawler | 11 | 4 |
| ChatGPT-User | OpenAI | Fetches when a user asks | 15 | 5 |
| MistralAI-User | Mistral | Fetches when a user asks | 14 | 5 |
| Perplexity-User | Perplexity | Fetches when a user asks | 12 | 4 |
| Claude-User | Anthropic | Fetches when a user asks | 11 | 6 |
| Applebot | Apple | Search engine | 10 | 3 |
| Bingbot | Microsoft | Search engine | 2 | 2 |
| Googlebot | Search engine | 1 | 2 |
Blocking training while staying in AI search
OpenAI, Anthropic, Google and Apple each publish separate tokens for training and for search, so a site can opt out of one and not the other. The data shows that sites use the split: most sites that block a training token leave the matching search token open.
| Blocks | But allows | Cited sites | Small-business sites |
|---|---|---|---|
| GPTBot | OAI-SearchBot | 33 of 43 | 42 of 46 |
| ClaudeBot | Claude-SearchBot | 26 of 36 | 37 of 41 |
| Google-Extended | Googlebot | 30 of 31 | 35 of 37 |
Sites with no rules, and sites that never answered
46 of the cited sites and 109 small-business sites answered /robots.txt with a 4xx error. A missing file means no rules, so every crawler may fetch every page. 11 cited sites and 4 small-business sites block every bot through "User-agent: *" with "Disallow: /", and most of the cited ones let Googlebot back in by name.
12 cited sites and 10 small-business sites never answered: a server error, a rate limit or a timeout on every attempt. Google treats a robots.txt that keeps failing with a server error as a block on the whole site, so for a crawler these sites can be as closed as one that disallows everything.
390 of the cited sites (39.5%) and 252 small-business sites (25.5%) name at least one AI bot in a group of its own, though most of those groups allow it.
What to do with your own robots.txt
Decide separately for training and for answers. Blocking GPTBot, ClaudeBot or Google-Extended keeps pages out of those operators' training data. Blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot or the user-fetch agents keeps them out of the answers themselves.
Then check that the file says what you decided. Copied block lists often catch search and user-fetch agents along with the training crawlers. And robots.txt is not the only gate: a CDN or firewall rule can block an AI crawler that robots.txt allows.
- List the AI bots you want to allow or block, by token, rather than relying on the wildcard group.
- If you block training crawlers, leave OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-fetch agents open unless you want to be absent from AI answers.
- Make sure /robots.txt answers quickly with a 200 or a 404. A file that times out or errors can read as a full block.
- Check your CDN's bot settings as well as the file.
Method
The cited cohort is the 1,000 hosts AI engines cited most often in RankEcho audit answers, leaving out the audited sites themselves and redirect or cache hosts. Each was cited between 8 and 1,033 times. The answers came from audits of 32 sites, and 39,771 of the 44,371 citations came from audits of rankecho.io itself, whose prompts ask about AI visibility and SEO software. So the cohort leans heavily towards marketing, SEO and software sites: read it as the sites AI engines cite in that market, not the web at large. The small-business cohort is 1,000 homepages from the public US business listings behind our website readiness screening. No domain is named here; only aggregate counts are published.
Each host's /robots.txt was fetched once, with up to three attempts, between 16 November 12:00 and 17 November 00:01 UTC, with the user agent RankEcho-AgentCheck/1.0. A file served as an HTML page counts as missing. A bot counts as blocked when the file disallows the site root for that token, either in its own group or through the wildcard group when it has none, read by the same parser RankEcho's audits use.
The kinds follow each operator's documentation. Amazonbot is listed apart because Amazon describes it as used for both search features and training.
Limits
robots.txt is declared policy. It does not show CDN or firewall rules, rules on paths below the root, or page-level meta tags, and it cannot show whether a bot obeys it. Read these numbers as what sites ask for, not as what crawlers can reach.
The cited cohort is selected by being cited, so it leans towards sites that let AI search crawlers in, and most of its citations come from one market, AI visibility and SEO software, where sites want to be cited. That makes its low block rates expected; it does not show that blocking causes fewer citations, and news publishers block far more often. The small-business cohort comes from US listings and is not a sample of the whole web. Both cohorts were read once, and robots.txt files change.
Frequently asked questions
In our November 2026 sample, 10.5% of 1,000 sites cited in our audits' AI answers (mostly marketing, SEO and software sites) and 5.4% of 1,000 small-business sites block at least one AI or search bot in robots.txt. Most of them block only training crawlers.
Among the cited sites, ByteDance's Bytespider (8.1%), then Common Crawl's CCBot and Amazonbot. Among small-business sites, OpenAI's GPTBot (4.6%).
Blocking GPTBot opts out of OpenAI's training crawls. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which have their own tokens. 33 of the 43 cited sites in our sample that block GPTBot leave OAI-SearchBot open.
Almost never: 1 cited site and 2 small-business sites blocked Googlebot. Google-Extended, the token for some Gemini uses, was blocked by 31 and 37, nearly all of which allow Googlebot.
Yes. The aggregate table is a CSV under CC BY 4.0; please link to this page. No domains are published.
