What llms.txt is for
llms.txt is a proposal, published at llmstxt.org in 2024, for a plain Markdown file that helps large language models understand a website. Web pages are built for people and browsers: navigation, scripts, cookie banners and layout markup all compete for space in a model's limited context window. An llms.txt file skips all of that and gives an assistant a short, curated overview plus links to the pages that matter most.
It is aimed at inference time, when an assistant or agent is answering a question about your product right now, rather than at training. It is most useful for documentation, developer tools, SaaS products and any site where accurate, up-to-date answers matter. The proposal also suggests offering clean Markdown versions of important pages, which you can link to from the file.
The llms.txt format
The file uses ordinary Markdown in a fixed order. It starts with a single H1 containing the site or project name, which is the only required part. Next comes a blockquote with a short summary of the key information needed to understand the rest of the file. After that you can add any number of paragraphs or lists with extra detail, but no headings.
The rest of the file is made of H2 sections, each containing a Markdown list of links. Every item is a link in the form [Title](URL), optionally followed by a colon and a short note. A section named Optional has a special meaning: the links in it can be skipped when a shorter context is needed, so put secondary or background pages there. This generator produces exactly this structure and escapes characters that would break the Markdown.
llms.txt, robots.txt and sitemap.xml compared
The three files do different jobs. robots.txt controls access: it tells crawlers which paths they may or may not fetch, and it is the place to allow or block AI bots. sitemap.xml is a machine-readable list of every URL you want search engines to discover, usually with last-modified dates. llms.txt is neither a permission file nor a complete index. It is a short, human-readable guide written for language models, pointing to your best and most current content.
That means llms.txt does not block or invite any crawler, and listing a page in it does not override a robots.txt rule. Keep all three: use robots.txt for access, the sitemap for discovery, and llms.txt to tell assistants where the authoritative answers live.
Should you expect results from it?
Be realistic. llms.txt is a community proposal, not a standard from a standards body, and support from AI providers is uneven and largely undocumented. Some developer tools, IDE assistants and agents read it when they are pointed at a site, and many documentation platforms now publish one automatically. Major AI search products do not document it as a ranking signal.
Because it takes minutes to create and costs nothing to host, it is a reasonable low-risk addition, especially for documentation and software sites. Keep it short, link to pages that are accurate and well written, and update it when your key pages change. A stale llms.txt that points to outdated docs does more harm than having none.
