Release,
Data reports: crawlers, llms.txt, citations
Two data reports on robots.txt and llms.txt at 2,000 sites, a larger retrieved-but-not-cited study, and seven alternatives pages rewritten.
What changed
Which AI crawlers do sites block?
robots.txt on 1,000 sites cited in AI answers in our audits, mostly marketing, SEO and software sites, and 1,000 small-business sites: block rates for 22 AI and search bots, and how many sites block training crawlers but let AI search in. See it
How many websites have an llms.txt?
/llms.txt on the same 2,000 sites: how many publish one, how many follow the proposal's shape, how long the files are, and how rarely AI crawlers fetched our own. See it
Retrieved but not cited, five times larger
1,329 ChatGPT and Claude API runs from 1 to 16 November: which of the 14,090 pages their search returned the answers cited, by place in the results, kind of page and set of questions. See it
Download the numbers
The aggregate tables behind every figure are CSVs under CC BY 4.0. No domains are named.
Alternatives pages rewritten
The AirOps, AthenaHQ, Goodie, HubSpot AI Search Grader, Scrunch, Surfer and Writesonic alternatives pages now list five tools each, with prices and the date we checked them. See it
