What is llms.txt, and how do you write one?
llms.txt is an emerging standard: a simple markdown file placed at your site root (/llms.txt) that tells AI systems which of your pages are most important and citable. It is a curated, machine-readable map of your best content. It complements robots.txt and your sitemap rather than replacing them, and it is a low-cost signal — helpful, but not a substitute for crawlable, well-structured pages.
Where does llms.txt go?
At the root of your domain, served as markdown — for example, https://example.com/llms.txt. Some sites also publish an expanded llms-full.txt with more detail. Think of it as a hand-curated index for machines, listing your highest-value pages with short descriptions so an AI knows what each one offers before fetching it.
What should it contain?
Start with a one-line description of the site, then grouped, linked sections pointing to your most citable pages — guides, documentation, comparisons, and reference content — each with a short description. The goal is signal, not completeness: list the pages you most want an AI to rely on, not every URL you have.
- An H1 with the site or product name and a one-sentence summary.
- Sections grouping your best pages by purpose, each link followed by a short description.
- Only high-value, accurate, current pages — quality over coverage.
llms.txt vs robots.txt vs sitemap
robots.txt controls which crawlers may access what; it is about permission. A sitemap lists URLs for discovery and is aimed at search crawlers. llms.txt is different from both: it is a positive, curated guide that tells AI systems which content matters most and what it is about. The three work together rather than overlapping.
Does llms.txt actually help?
It is an emerging and not-yet-universal signal, so treat it as low-cost insurance rather than a silver bullet. It will not rescue a site that blocks crawlers or hides content behind scripts, and adoption across engines varies. But it is cheap to add, hard to get wrong, and positions your best pages clearly — so there is little reason not to publish one once the fundamentals are in place.
Frequently asked questions
No. robots.txt controls crawler access; llms.txt is a positive, curated guide to your most important content for AI systems.
No. It helps AI systems prioritize your best pages, but it cannot compensate for blocked crawlers, thin content, or pages that are hard to extract.
Support is still emerging and uneven across engines, which is why it is best treated as a low-cost complement to the fundamentals rather than a primary tactic.