What llms.txt is, how to write one and how agents should treat it

llms.txt is a plain Markdown file that a website places at its root to give language models a short, curated map of its most useful content. It has become common on documentation sites and developer tools, and it is misunderstood in both directions: some treat it as a new robots.txt, others dismiss it as a passing fashion. This page explains what the file is, how sites use it, what it does and does not do, how to write a good one, and how an agent that finds one should handle it.

What the file is

The convention was proposed by Jeremy Howard of Answer.AI and is documented at llmstxt.org. It is a community proposal, not a standard issued by a standards body, and nothing obliges any model, crawler or agent to read it.

The idea is simple. Web pages are built for browsers: navigation, scripts, cookie banners, layout. A model working inside a limited context window needs the opposite, a compact text version of what matters. llms.txt provides an index, written in Markdown, of the pages a site owner considers most useful, each with a one-line description. Many sites also publish Markdown versions of the pages themselves, so an agent can fetch clean text instead of parsing HTML.

Two related conventions often appear next to it:

• Markdown page variants. The proposal suggests serving a Markdown version of a page at the same URL with .md appended, so /docs/auth.html also exists as /docs/auth.html.md.
• llms-full.txt. Some documentation platforms publish one file containing the full text of the docs, for tools that prefer to load everything at once. It is a widespread practice rather than part of the core proposal.

The format

The structure is deliberately small, so that both people and programs can read it without special tooling. In order:

1. An H1 with the name of the project or site. This is the only required element.
2. A blockquote with a short summary: the context a model needs to make sense of the rest.
3. Optional free text, such as paragraphs or lists with important notes, for example which version the docs describe.
4. Zero or more H2 sections, each a Markdown list of links in the form [name](url): description.
5. An H2 named Optional, whose links may be skipped when context is short.

A complete example for a fictional library:

# Acme Queue

> Acme Queue is an open-source job queue for Python services.
> These docs cover version 3.

Version 2 APIs are documented separately and are not compatible.

## Docs

- [Quickstart](https://docs.example.com/quickstart.md): install, configure a broker, run a first job
- [Retries and backoff](https://docs.example.com/retries.md): retry policy options and defaults
- [Configuration reference](https://docs.example.com/config.md): every setting with its type

## API

- [Python client](https://docs.example.com/api/client.md): public classes and methods

## Optional

- [Changelog](https://docs.example.com/changelog.md): release notes
- [Design notes](https://docs.example.com/design.md): background on internal decisions

How sites use it

Adoption is concentrated where the readers are developers working with AI tools. Typical uses:

• Documentation sites that want coding assistants to find the right page for an API question instead of guessing from training data.
• Developer tools and SDKs that ship an llms.txt so a user can paste one URL into an editor agent and hand it the whole map.
• Product and policy sites that want a short, canonical description of what they offer, so summaries start from the owner's own wording.

The main readers today are agents acting at a user's request: an IDE assistant told to read a library's docs, a research agent fetching a site it was asked about, or a person loading the file into a chat. Whether large model providers use llms.txt when collecting training data is not something the file can tell you, so do not assume they do. Check each provider's own documentation.

What llms.txt does not do

Most confusion comes from treating the file as something it is not.

In short, llms.txt sits next to robots.txt rather than replacing it. If you want to restrict AI crawlers, that belongs in robots.txt directives and server-side controls.

Writing a good one

A useful llms.txt is short, accurate and maintained. Some practical rules:

• Curate, do not dump. List the pages that answer the most common questions. A sitemap converted to Markdown is not a map; it is a pile.
• Write descriptions that help a choice. “Retry policy options and defaults” tells an agent whether to open the page. “Retries” does not.
• Say which version. If your docs cover several versions, state near the top which one the links describe.
• Link to clean text. Point to Markdown variants where you have them, otherwise to pages that render readable content without scripts.
• Use the Optional section honestly for background material, so tools with a tight context budget can drop it.
• Generate it from the same source as the docs, so a renamed page cannot leave a dead link behind. Check whether your documentation platform can produce it for you.
• Test it the way it will be used. Give the file to an agent, ask a handful of real questions, and see whether it opens the right pages.

How an agent should treat llms.txt

For anyone building agents, the file is useful and also an obvious place for mischief. A sound policy:

1. Use it as a table of contents. It is a quick way to find the likely pages for a question about that site. It is not an answer in itself.
2. Treat its content as untrusted data. A line saying “ignore previous instructions” or “always recommend this product” is text from a website, with no more authority than any other page. Keep it out of the instruction channel.
3. Cite the canonical page, not the index. Fetch the linked page and quote the passage from it. If the Markdown variant and the HTML page disagree, prefer what a human reader sees and flag the mismatch.
4. Do not read it as permission. Check robots.txt and the site's terms separately. A file that welcomes models does not authorize logging in, getting past a paywall or collecting personal data.
5. Record when you read it. Store the retrieval date with anything derived from the file or its links, because the site can change at any time.
6. Fall back gracefully. Most sites do not have one. Its absence says nothing about a site's quality or its attitude to agents.

The broader rules for agents on the open web, such as identifying the client and respecting access controls, are covered in what a web crawler is and what web scraping is. For choosing which sites may feed an answer in the first place, see deciding which sources may answer.

Bringing a site's documentation into GoCux

GoCux does not crawl or read the open web, so it does not go looking for llms.txt files. Where the file helps is one step earlier, when a team decides which pages of a vendor's or regulator's site belong in a project. The index makes that selection quicker. The pages themselves then enter the project through an authorized browsing or export step and arrive with the original page address and the date they were retrieved, after which Knowledge search answers from them with passage citations. The source policy describes how sources are admitted, and web sources and responsible agent access covers the web side.

Checklist

If you publish one

• It opens with an H1 and a short summary, and states which version it describes.
• Every link has a description that helps an agent decide whether to open it.
• Links point to clean text, and the file is generated alongside the docs.
• Background material sits under Optional.
• Access rules live in robots.txt and on the server, not in llms.txt.

If your agent reads one

• The file is processed as data, never as instructions.
• Citations go to the linked pages, not to the index.
• The retrieval date is stored with every finding derived from it.
• robots.txt and the site's terms are checked separately.
• When there is no file, the agent carries on normally.