An llms.txt file makes your site eligible for AI assistants to use. It does not get you cited.
Builders comparing what they did to get mentioned by AI engines keep arriving at that line in almost the same words: the file work makes you eligible, and mentions elsewhere are what get you cited. So publish the file so an agent can find your best content in one read, keep it current as you publish, and treat citations as separate work.
What Is Llms.txt
An llms.txt file is a proposed standard: a Markdown file at the root of a site, or at any subpath, that gives AI crawlers and language models a curated summary of the site's most important content. It works as a map, pointing an agent to high-value pages instead of leaving it to wander raw HTML.
The proposal, published at llmstxt.org, starts from a practical problem. Web pages are built for people and wrapped in navigation, ads, and scripts, and turning them back into clean text is hard and imprecise. Whole sites rarely fit in a model's working memory, and every wasted token costs time and money. Agents do best with concise, expert information gathered in one place.
Llms.txt vs Robots.txt vs Sitemap.xml
The three files coexist and do different jobs.
- robots.txt declares what automated access is acceptable. It is about blocking.
- sitemap.xml lists every indexable URL, without context. It is usually far too large for a model's context and full of pages an assistant does not need.
- llms.txt is a small, curated selection with context, built to be read at the moment a model is answering a question.
The short version: a sitemap lists everything for search engines, and llms.txt highlights the best of it for AI. It is an inclusion file, not a blocking file.
Decide What to Feature
Before writing the file, decide which pages deserve a place on the map. At minimum, include the pages that tell an agent what you do and how you help:
- product or service pages;
- current posts or documentation;
- pricing;
- about and contact.
Link to clean, model-friendly versions where you have them, such as Markdown copies of pages, and keep the file itself short enough to fit in context. Detail lives behind the links and is fetched only when needed. A file that lists everything is a second sitemap; a file that maps the pages that matter is useful.
How to Shape the Markdown
The format is specific, and the order matters:
- an H1 with the project or site name, the only required part;
- a blockquote with a short summary holding what a reader needs to understand the rest;
- optional plain paragraphs with background and guidance;
- optional H2 sections of links, each item a link plus an optional note after a colon.
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)
A section headed Optional holds secondary links by convention, ones an agent can skip when it is short on space. When several files apply to a page, agents use the most specific one. Pages can also point to their own Markdown copy with rel="alternate" type="text/markdown" and to the llms.txt that covers them with rel="describedby", either as HTML link elements or as HTTP Link headers.
Write for parsing as well as reading: plain language, a short informative note on every link, no unexplained jargon. Then test it by asking an agent questions about your content with only the file as its starting point.
What Real Files Look Like
Real files vary widely and still work. Hugging Face publishes a comprehensive knowledge base with headings and code. Vercel uses titled, described sections. Zapier and Cal.com run flat lists of links with short notes. As long as the Markdown is valid, the variation is normal.
Some products go further and make the file the whole interface. qequals, an API for Google suggestions and live results, has no docs site: its llms.txt is the contract, explaining how to mint a key, the two calls, the JSON shapes, the error codes, and the quota headers, so an agent can go from discovery to a working call in one read. That is the pattern at its purest, one file written for the reader that will actually use it.
Where to Put It
A file covering the whole site goes in the root, so it loads at /llms.txt. A file covering only documentation goes in that folder, for example /docs/llms.txt for everything under /docs/. A file describes the path it sits on, so a project that controls only one path can still publish one.
After uploading, open the address directly and confirm it loads as plain text.
Keep It Current as You Publish
The builders who take this seriously treat the file as part of publishing, not a one-time task. One WordPress workflow regenerates both llms.txt and a fuller companion, llms-full.txt, to the standard every time a post goes out, and serves a clean Markdown copy of each post at its own address with the theme's HTML stripped away. Another builder's setup pings Bing on every publish and keeps robots.txt, llms.txt, and llms-full.txt maintained side by side.
The habits worth copying:
- review the links on a schedule, removing dead pages and adding new ones;
- keep the llms-full.txt companion in step with the short file if you publish one;
- keep clean Markdown copies of posts or docs, so the map points at readable pages.
A stale map sends agents to dead ends. The file earns its keep only while it matches the site.
How to Confirm It Reads
Most site builders and documentation platforms now generate the file for you: Mintlify, GitBook, Yoast SEO, AIOSEO, and Wix among them. Whoever writes it, check it three ways:
- run an llms.txt checker or validator against the live address to catch format and placement errors;
- run Chrome Lighthouse, whose agentic-browsing checks look for the file;
- ask an agent a question about your content starting only from the file, and see whether it follows the right link.
Does Llms.txt Work Yet
The honest answer is: not much, so far.
NerdyData counts reported through Semrush found only 951 domains with a published file as of July 2025. Google's John Mueller has said no AI system currently uses llms.txt. A Semrush test written up by Search Engine Land checked server logs from mid-August to late October 2025 and found zero visits to the file from GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, with only a few ordinary crawler hits treating it like any other page. SEO forum discussion on Reddit reaches the same conclusion: the major crawlers do not rely on it yet.
The signals on the other side are real but narrow. Thousands of sites publish one, documentation platforms create it automatically, Lighthouse audits for it, and AI labs including OpenAI, Anthropic, and Google publish their own for their developer docs. Semrush's own advice lands in the middle: probably not worth much time right now unless you want to experiment.
Eligible Is Not Cited
Read the file as eligibility plus a clean entry point for agents. It makes your site legible to an agent that comes looking. It does not make the agent come, and it does not make an answer quote you.
That part happens elsewhere. People comparing notes say models cite forum threads, best-of lists, and reviews more often than a brand's own site, and that you cannot improve what you do not track. Whether an AI engine actually mentions you is a separate thing to measure, with its own prompts, engines, and cadence. Publish the file, keep it current, and measure the citations on their own.
