llms.txt is a proposed standard for a markdown file at the root of your domain that tells large language models what your site contains and where the good stuff lives. It has been written about enthusiastically for two years. Most of that writing oversells it.
Here is the short version. llms.txt is cheap to create, carries no risk, and is genuinely useful in one specific context. It is not read by the major AI search crawlers, Google has said outright that it does not support it, and it will not affect your rankings or your citation rate. Both of those things are true at once, and the interesting question is what to do about it.
What the proposal actually is
Jeremy Howard of Answer.AI proposed llms.txt in September 2024. The problem it addresses is real: HTML pages are full of navigation, scripts and boilerplate, context windows are finite, and an agent trying to understand a site has to burn tokens parsing markup to find the content.
The proposal is a single markdown file at /llms.txt that acts as a curated index. Not a full copy of your site - a map, with links to clean markdown versions of the pages that matter, plus enough description that a model can decide what to fetch.
It is deliberately modelled on robots.txt and sitemap.xml in placement, but it is nothing like them in function. Robots.txt restricts. Sitemap.xml enumerates everything. llms.txt curates and explains.
There is also a companion convention, llms-full.txt, which concatenates your entire documentation set into one large markdown file for direct paste-in or single-fetch consumption.
The file format
The spec is short and mostly optional. In order:
- An H1 with the site or project name. This is the only required element.
- An optional blockquote immediately after it, giving a one-or-two-sentence summary of what the project is.
- Optional plain paragraphs with any additional context a model needs - scope, conventions, things it should not assume.
- Any number of H2 sections, each containing a bulleted list of markdown links. Each bullet is a link, optionally followed by a colon and a short description of what is behind it.
- A final H2 named
Optional, which has special meaning: everything listed under it can be skipped when context is tight.
So a real file opens with something like # Acme API, then > REST API for scheduling and dispatch, with SDKs for Python and TypeScript., then sections such as ## Getting started and ## Reference, each holding bullets in the form `- Authentication: API keys, OAuth flows, and token rotation.`
The links should point to clean markdown where possible. A related convention that grew up alongside llms.txt is serving a .md version of any documentation URL - append .md and get the raw markdown. That pairing is where the format does its best work.
There is no validator, no registry, and no verification step. You write the file, you upload it to your root, that is the whole mechanic.
Does anything actually read it?
This is the part most articles skip. The evidence is not encouraging.
Adoption is low. A study of the top 1,000 sites found roughly 8.7% had an llms.txt file. A broader study across 300,000 domains found around 10.1%. After two years of promotion, roughly nine sites in ten have not bothered - and that number is inflated by documentation platforms that generate the file automatically without anyone asking.
Consumption is lower still. In one analysis of over 500 million AI bot visits, only a few hundred requests fetched llms.txt at all. Not a few hundred thousand. A few hundred. The crawlers building the indexes behind ChatGPT, Perplexity and Claude are, in practice, fetching your HTML like everyone else.
No major model provider has publicly confirmed using llms.txt for retrieval, ranking or citation selection. Absence of confirmation is not proof of absence, but combined with the fetch data it is about as clear as this kind of evidence gets.
The uncomfortable comparison is the keywords meta tag: a well-intentioned standard that asked publishers to self-describe, that search engines declined to trust because self-description is trivially gamed, and that quietly became irrelevant. llms.txt has the same structural weakness. A file you write about yourself, with no verification, is a weak signal by construction.
Google has said no
In 2025, Gary Illyes of Google stated publicly that Google does not support llms.txt. Google's position has been consistent since: the crawling infrastructure behind AI Overviews and AI Mode is Googlebot, working from the same index as Search, and there is no separate llms.txt ingestion path.
That matters more than it might seem, because Google is the largest single source of AI-surfaced answers by volume. If Google does not read it, the file cannot be a general-purpose AI SEO tactic, whatever else it might be good for.
Take anyone selling an "llms.txt generator" as a ranking service at a discount accordingly. The file takes minutes to produce. It is not a growth lever.
Where llms.txt genuinely is used
None of the above makes the format useless. It makes it useful somewhere other than where it is usually marketed.
llms.txt works in pull contexts - where a human or an agent has already decided to read your documentation and needs the fastest path through it. It does not work in push contexts, where a crawler is deciding on its own what to index.
The pull contexts are real and growing:
Coding agents and AI-native IDEs. When a developer points Cursor, Claude Code, Windsurf or a similar tool at your library's docs, an llms.txt gives it a curated entry point instead of a rendered marketing site. Several of these tools support adding documentation by URL, and a good llms.txt measurably improves what the agent produces. If you ship a developer tool, this is the strongest argument for the file by a wide margin.
Documentation platforms. Mintlify, Docusaurus and others generate llms.txt and llms-full.txt automatically, which is why adoption skews so heavily toward developer docs. If you are on one of these, you may already have the file.
MCP servers and in-product assistants. When you build an assistant over your own docs, or expose your documentation through an MCP server, llms.txt is a clean, stable manifest of what exists and what it is for. You control both ends, so the trust problem disappears.
Direct paste. People genuinely copy llms-full.txt into a chat to give a model your entire documentation set in one move. Unglamorous, and one of the most common actual uses.
The pattern is consistent. llms.txt earns its keep when something has been explicitly told to read you. It does nothing when something is choosing whether to read you at all.
How to write a good one
If you are doing it, do it properly - a bad llms.txt is worse than none, because the failure mode is an agent confidently working from stale information.
Curate hard. The value is in what you leave out. Twenty well-chosen links beat two hundred. If your file is just your sitemap in markdown, you have built a worse sitemap.
Write descriptions for a model, not a browser. The text after each colon should say what is in the document and when it applies, not sell it. "Rate limits, retry semantics, and error codes for the v2 API" is useful. "Everything you need to succeed with our powerful API" is noise.
Point at clean markdown. If you can serve a .md variant of each linked page, do it. If you cannot, link the HTML - the format still works, just less efficiently.
Use the Optional section honestly. Put genuinely secondary material there - changelogs, deep reference, background reading. It exists so a model can degrade gracefully under a tight context budget.
Keep it in your build. The single biggest failure is drift. An llms.txt written by hand in one afternoon is stale within a quarter. Generate it from the same source as your navigation, or accept that it will rot.
Assume it is public. It is a file at your web root, indexable and readable by anyone. Nothing goes in it that you would not publish on a normal page.
If you want a starting point rather than a blank file, the Fix Pack in the free SEO audit generates a ready-to-use llms.txt from your existing site structure, which you then edit down - editing is faster than authoring, and the curation is the part that needs your judgement anyway.
Is it worth your time?
Yes, with the expectations set correctly.
Do it if you have developer documentation, an API, or a product that AI coding tools will be asked about. The payoff there is concrete and does not depend on any crawler adopting anything.
Do it if it costs you an hour or is generated automatically. There is no downside beyond that hour, no ranking risk, and a plausible chance that consumption improves as agent traffic grows.
Do not do it expecting rankings, citations or AI-search visibility. The evidence says it will not deliver those, and Google has said so directly.
Do not do it instead of the work that does move AI visibility. Server-side rendered content that crawlers can read without executing JavaScript, accurate structured data, a clean internal link graph, and pages that answer a question in the first paragraph - those determine whether you get cited. Checking how assistants currently describe your site is a better use of the same hour than polishing a file almost nothing fetches, and if you want to watch how AI crawlers actually behave on your domain, your own server logs will tell you more than any proposal document.
Write the file. Ship it in fifteen minutes. Then go fix your rendering.
Frequently asked questions
Does llms.txt improve AI search rankings?
No. There is no evidence that any major AI search system uses llms.txt for ranking or citation selection, and Google stated publicly in 2025 that it does not support the file. In one study of more than 500 million AI bot visits, only a few hundred requests fetched llms.txt at all.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a curated index - an H1, a summary, and sections of annotated links to your most important pages. llms-full.txt is your entire documentation set concatenated into a single markdown file, intended to be fetched or pasted whole. Use llms.txt as the map and llms-full.txt when something needs everything at once.
Do I still need robots.txt and sitemap.xml if I have llms.txt?
Yes. They do unrelated jobs. Robots.txt controls crawler access and is respected by every compliant bot, sitemap.xml tells search engines which URLs exist, and llms.txt is an optional curated summary that most crawlers ignore. llms.txt replaces neither.
Where do I put the llms.txt file?
At the root of your domain, served as plain text or markdown over HTTPS, so it resolves at yoursite.com/llms.txt. Do not put it in a subdirectory, do not require authentication, and make sure it is not blocked in robots.txt - the whole point is that anything can fetch it without negotiation.