llms.txt Explained — Does It Actually Help?

Short answer: llms.txt is a proposed convention: a plain Markdown file at the root of your site that lists your most important pages with short descriptions, intended as a guide for AI systems. It is cheap to add and harmless. There is little public evidence that major AI assistants rely on it today, so treat it as a low-cost extra and not as a strategy.

Table of Contents

1. What llms.txt Is
2. How It Differs From robots.txt and a Sitemap
3. What It Looks Like
4. Does It Work?
5. How We Generate Ours
6. What Matters More
7. Key Takeaways

What llms.txt Is

The proposal addresses a real difficulty. A web page is full of navigation, scripts, and layout, which makes it awkward for a language model to extract the content. A site may also have thousands of pages, more than fits in a model's working context.

llms.txt offers a curated shortcut: a single file, in Markdown, that says what the site is and links to the pages that matter, each with a one-line description. Some sites also publish a longer companion file containing full content.

It is a community proposal, not a formal standard, and adoption is voluntary on both sides.

How It Differs From robots.txt and a Sitemap

Two misunderstandings are common.

• llms.txt does not control access. It cannot block or allow a crawler. That is the job of robots.txt, and of your CDN and firewall.
• llms.txt does not replace a sitemap. A sitemap is exhaustive. llms.txt is selective.

What It Looks Like

  # Example Company

  > Example Company makes scheduling software for clinics.

  ## Product

  - [Features](https://example.com/features): What the product does
  - [Pricing](https://example.com/pricing): Plans and prices

  ## Documentation

  - [Quickstart](https://example.com/docs/quickstart): Set up in ten minutes

  ## Guides

  - [How to reduce no-shows](https://example.com/blog/reduce-no-shows): Practical steps for clinics

A title, a one-sentence summary, then sections of links with brief descriptions.

Does It Work?

Honestly: it is not yet clear.

• The major AI companies have not, as far as public statements go, committed to using llms.txt as an input for answers or training.
• Site owners who check their logs often find few requests for the file from AI crawlers.
• Some developer tools and coding assistants do use it, particularly for documentation.
• Because it is plain text with links, it is at worst another crawlable page describing your site clearly.

So the reasonable position is: add it, because it costs almost nothing, and do not expect it to change your visibility by itself. Check your server logs to see whether anything requests it.

How We Generate Ours

Our first llms.txt was a static file with three links. It was accurate when written and out of date within weeks, because nobody remembers to edit a text file when a page is added.

We replaced it with a generated route. The file is now built from the same content modules that render the pages themselves: documentation, features, comparisons, industries, and blog articles. When a page is added to the site, it appears in llms.txt automatically. That took the file from three links to well over a hundred.

Two things we learned:

• Generate it from your source of truth. A hand-maintained file drifts.
• Do not keep a static copy as well. A static file and a generated route at the same path collide at build time. One of them has to own the URL.

If your site is built from a content system or structured data, generating llms.txt alongside your sitemap is a small job.

What Matters More

If your goal is to be cited by AI assistants, these come first.

1. Crawler access. Check the robots.txt that is actually served, not the one in your repository. A CDN may be adding rules (how to check).
2. Answer-first content. Pages that state the answer in the opening sentences (how to write it).
3. Plain positioning. A consistent statement of what you are.
4. Mentions elsewhere. Reviews, comparisons, and discussions on other sites.
5. Structured data. Facts in machine-readable form (schema guide).

llms.txt belongs after all of these.

Key Takeaways

• llms.txt is a curated guide, not an access control or a sitemap.
• Evidence that major assistants use it is thin.
• Generate it from your content so it stays current.
• Crawler access and clear content matter far more.

Conclusion

llms.txt is a reasonable ten-minute addition and a poor place to spend a week. Add it, generate it automatically, and put your real effort into being reachable and worth citing.

Read about AI crawlers.