Does llms.txt work? What 30 days of AI crawler logs show

By Abiot Y. Derbie - Founder, RankEcho - Updated 2026-11-03

Key findings

These are one site's logs for one month: rankecho.io, a young site, as Cloudflare recorded every visitor request from 4 October 2026 to 3 November 2026. They show what the bots did, not what they would do on another site.

• AI bots rarely ask for llms.txt. Verified AI bots fetched it 8 times in 14,806 requests, about once in every 1,851. No verified AI bot fetched it more than three times in the month.
• robots.txt is what crawlers check. The same AI bots fetched robots.txt 546 times and sitemaps 259 times. ClaudeBot alone fetched robots.txt 206 times and the sitemaps 205 times, and llms.txt once.
• llms.txt was treated like any other page. A page on rankecho.io was fetched by verified AI bots a median of 3 times in the month; llms.txt, 8 times by all of them together.
• SEO tools read it more than AI bots did. Of the 68 requests for /llms.txt, 23 came from SEO and site-audit tools and 8 from verified AI bots.
• The pages it lists were found without it. Verified AI bots fetched 218 of the 223 pages linked from rankecho.io's llms.txt during the month, through links and sitemaps.
• Many "AI bots" are not. 1,854 requests used an AI bot's name without Cloudflare verifying it, and 366 of them asked for files like /.env or credentials.json, which no crawler needs.

Why look at logs

llms.txt is a proposed Markdown file at the root of a site that lists its most useful pages for language models. The proposal is voluntary, and Google has said it does not use the file for Search. Most articles about llms.txt argue from that, for or against.

A site's own logs answer a narrower question that matters first: does any AI bot actually ask for the file? If none does, it cannot change what the bot sees. rankecho.io serves an llms.txt, 223 links today, and every verified bot that asked for it in this window got it. Its traffic goes through Cloudflare, which identifies the major AI bots and checks that a request really comes from them.

The data

The figures come from Cloudflare's analytics for rankecho.io (and www.rankecho.io), read through its GraphQL API on 3 November 2026. They cover 86,530 visitor requests over 30 days. Requests the site's own code made to itself, such as cache fills, are left out.

A request counts as a verified bot when Cloudflare has matched it to the bot's operator, not just to the name in its user agent. AI bots are those Cloudflare files as AI crawlers, AI search or AI assistants, plus OpenAI's OAI-SearchBot, which it files as a search engine crawler. Cloudflare did not verify any PerplexityBot request in this window, so Perplexity is not in the verified counts.

Cloudflare samples its analytics in busy periods, so 24,535 of the requests are estimates from samples. Every request for /llms.txt but one group of 14 was recorded unsampled.

Finding 1: AI bots check robots.txt and sitemaps, not llms.txt

Verified requests by bot, with how many asked for each of the three files a crawler might look for. Pages are everything else.

Amazonbot made almost half of all verified AI bot requests, and fetched robots.txt and llms.txt once each. ClaudeBot and OAI-SearchBot checked robots.txt often; GPTBot went through the sitemaps. Every bot that fetched llms.txt did so once, except Applebot, three times on one day.

Finding 2: who did fetch llms.txt

/llms.txt was requested 68 times in the month by anyone, against 2,715 requests for robots.txt and 744 for sitemaps. rankecho.io does not publish an /llms-full.txt; it was asked for 4 times, all by the same unnamed script.

SEOJuice's crawler fetched llms.txt on four different days, more than any AI bot. The likeliest reading is that llms.txt is checked today mostly by tools that grade sites for AI readiness, and by people looking at the file.

Finding 3: a bot's name in the log proves little

Anyone can put "ClaudeBot" or "GPTBot" in a request. 1,854 requests named one of the AI bots in this study without Cloudflare verifying them. 366 of those (20%) asked for secret files such as /.env, /.git or credentials.json, which is what break-in scanners do. ClaudeBot, for example, had 853 verified requests and 114 unverified ones, 70 of them for secret files.

PerplexityBot shows the other side: 947 requests carried its name and Cloudflare verified none, so real and fake cannot be told apart from this log. If you read your own server logs for AI bots, check the requests against the IP ranges each operator publishes before you count them.

So does llms.txt work?

For getting AI engines to find or read your pages, there is no sign of it here. The AI bots that crawl rankecho.io find pages through links and sitemaps, check robots.txt for what they may fetch, and almost never ask for llms.txt. A file that is fetched once a month cannot be steering what a crawler reads.

That is not a reason to delete it. An llms.txt costs little to keep, some coding assistants and documentation tools read it when a person asks them to, which would not look like a crawler in these logs. Keep it accurate if you have one. But do the work that crawlers demonstrably use first: a robots.txt that lets the AI bots in, a complete sitemap, and pages whose content is in the HTML.

• Check robots.txt allows the AI bots you want, and that your CDN or firewall does not block them anyway.
• Keep sitemaps complete and their dates true. GPTBot and ClaudeBot read them repeatedly here.
• Put each page's main content in the HTML the server sends, not only in JavaScript.
• If you keep an llms.txt, list only canonical, public pages and update it when pages move.

How to check your own logs

On Cloudflare, query the GraphQL Analytics API for the zone (the httpRequestsAdaptiveGroups dataset), filter on the path /llms.txt, and group by user agent and verified bot category. On the free plan it reaches back about a month. Compare the count with the same query for /robots.txt.

On your own server, search the access log for requests to /llms.txt and note the user agents. Treat a bot name as unverified until its IP address matches the ranges its operator publishes, and leave out requests for secret files: they are scanners, not crawlers.

Limits

This is one site over one month. rankecho.io is young and small, and bots may behave differently on a large or long-established site, or one whose llms.txt is linked from elsewhere. Some bots may read llms.txt only after a site reaches a size or popularity this one has not.

Cloudflare's counts in busy periods are estimates, and bots are told apart by their user agent and Cloudflare's verification, which covers some operators and not others. The logs show which files were requested, not what the bot did with them or whether any answer used them. RankEcho will repeat the count as the site grows.

Frequently asked questions