RankEcho

Release,

Data reports: crawlers, llms.txt, citations

Two data reports on robots.txt and llms.txt at 2,000 sites, a larger retrieved-but-not-cited study, and seven alternatives pages rewritten.

What changed

Which AI crawlers do sites block?

robots.txt on 1,000 sites cited in AI answers in our audits, mostly marketing, SEO and software sites, and 1,000 small-business sites: block rates for 22 AI and search bots, and how many sites block training crawlers but let AI search in. See it

How many websites have an llms.txt?

/llms.txt on the same 2,000 sites: how many publish one, how many follow the proposal's shape, how long the files are, and how rarely AI crawlers fetched our own. See it

Retrieved but not cited, five times larger

1,329 ChatGPT and Claude API runs from 1 to 16 November: which of the 14,090 pages their search returned the answers cited, by place in the results, kind of page and set of questions. See it

Download the numbers

The aggregate tables behind every figure are CSVs under CC BY 4.0. No domains are named.

Alternatives pages rewritten

The AirOps, AthenaHQ, Goodie, HubSpot AI Search Grader, Scrunch, Surfer and Writesonic alternatives pages now list five tools each, with prices and the date we checked them. See it

More releases