In June 2026, Google updated its AI optimisation documentation to say something unusually direct: llms.txt won't help your Google Search visibility, and it won't hurt it either. Search simply ignores the file.
Five weeks before that, Chrome shipped Lighthouse 13.3 with an llms.txt audit turned on by default.
So Google is simultaneously telling you the file is pointless and auditing your site for it. A lot of SEO coverage has treated that as a contradiction and picked a side. It isn't a contradiction. Those are two different Google teams answering two different questions, and once you see which question each one is answering, the whole llms.txt argument stops being confusing.
Here's the part that actually matters. Stripe ships an llms.txt. So do Vercel, Cloudflare, Anthropic and Mastercard. These are not companies that add files to their web root because a blog post told them to. They know Google ignores it. They ship it anyway, because they're building for an audience that isn't Google Search.
I've had this conversation with three clients this year. Every one of them arrived worried they were behind. Every one of them left having decided something different, because the right answer depends entirely on who visits your site and why.
What an llms.txt file actually is and how it differs from robots.txt and your sitemap. Exactly what Google said in June 2026, and what it didn't say. The adoption and effectiveness data, including the studies that found nothing. Whether your site is one of the ones that should ship it, and how to write one properly.
What is llms.txt?
An llms.txt file is a Markdown file placed at your site's root that gives AI systems a curated map of your most useful content. It lists your key pages with short descriptions, in priority order, so a model or agent can find what matters without crawling everything. It's a community proposal, not an official web standard.
Jeremy Howard proposed it in September 2024. Version 2 of the spec landed on 10 August 2026, incorporating two years of feedback from people actually using the thing.
The comparison people reach for is robots.txt, and it's the wrong one. Robots.txt is exclusionary. It tells crawlers where they may not go. Your XML sitemap is exhaustive. It lists everything you'd like indexed. llms.txt is neither. It's selective and editorial. You're saying "of the four hundred pages here, these twelve are the ones worth reading, and here's why."
That distinction matters because it changes who the file is for. A sitemap is written for a crawler that will visit every URL regardless. An llms.txt is written for something with a limited budget and a question, which describes an AI agent far better than it describes Googlebot.
The file sits at yourdomain.com/llms.txt. Plain Markdown, UTF-8, no special encoding required. You can view any site's file by typing that path into your browser, which is a genuinely useful way to see how companies you respect are approaching it.
| llms.txt | robots.txt | sitemap | |
|---|---|---|---|
| Purpose | Curate for AI | Block crawlers | List pages |
| Status | Proposal | Standard | Standard |
| Google uses it | No | Yes | Yes |
| Format | Markdown | Text | XML |
| Scope | Selective | Directive | Exhaustive |
| Upkeep | Manual | Low | Automated |
What Google actually said, and what it didn't
Google's position has been consistent since 2024, and the June 2026 update didn't reverse it. It softened the tone.
The older guidance discouraged the file broadly. The June update narrows "not needed" to Google Search specifically, while acknowledging that other AI services exist and might use it. Google filed the change in its documentation changelog as a clarification rather than a policy shift, and Search Engine Journal read it the same way.
John Mueller had been blunter earlier. He compared llms.txt to the keywords meta tag, which is about as damning a comparison as exists in SEO. At Search Central Live Deep Dive Asia Pacific, Gary Illyes and Amir Taboul confirmed Google wasn't pursuing it.
The logic behind that position is sound, and it's worth internalising beyond this one file. A signal you write about your own site can't be trusted to rank your own site. That's precisely why the keywords meta tag died. Anything self-declared is gameable in about ten minutes, so it never becomes a ranking lever. Expect llms.txt to follow the same path with Search.
But notice the boundary of the claim. Google said llms.txt does nothing for Google Search. It did not say the file does nothing at all, and it does not speak for OpenAI, Anthropic, Perplexity or the coding agents that increasingly sit between your content and your reader.
The adoption data, read honestly
This is where most coverage falls apart, because the studies disagree and people quote whichever number supports their headline.
SE Ranking analysed roughly 300,000 domains and found about 10.1% had adopted the file. They then ran a statistical model and an XGBoost classifier to test whether having an llms.txt correlated with AI citations. The result was flat. No effect. Removing the variable actually improved model accuracy.
Ahrefs came at it from the read side. Across 137,210 domains they tracked, 97% of llms.txt files received zero bot requests during May 2026. Their sample skews technical, so treat that as an upper bound on real-world adoption rather than a representative slice of the web.
Then there's the detail that should end the fear-of-missing-out argument entirely. SE Ranking's adoption rate barely moved across traffic tiers: 9.88% for low-traffic sites, 10.54% for mid, 8.27% for high. High performers aren't quietly doing this while everyone else sleeps. If anything, the biggest sites are the least interested.
Different panels produce wildly different numbers depending on what you sample. A curated 219-host panel measured in August 2026 put adoption above 50%. A tracker of the Tranco top 1,000 put it near 9%. Both are correct. One sampled the sectors whose readers run coding agents. The other sampled the web by traffic. When someone quotes you an llms.txt adoption figure, ask what they sampled.
And the governance point nobody mentions: llms.txt has no backing from the W3C, the IETF or any recognised standards body. As of early 2026, no major AI company had publicly committed to reading or acting on it in production. GPTBot fetches it occasionally. Fetching is not the same as using.
So who should actually ship one?
Here's my honest read after arguing about this with clients all year.
This is the real use case, and it's why Stripe and Vercel bother. When a developer asks a coding agent to integrate your API, that agent needs a fast route to the right docs. Platforms like Mintlify and GitBook generate the file automatically, so the cost is close to zero.
A small site with a stable set of important pages loses nothing by having a clean twelve-line file at the root. Call it insurance. Fifteen minutes, no ongoing maintenance burden.
This is the decision most people actually face, and llms.txt loses that trade every time. If your Core Web Vitals are failing or half your pages aren't indexed, an llms.txt is rearranging cushions on a sofa that's on fire.
Popular approach, bad one. You're creating a parallel content architecture to maintain, with all the duplication and drift that implies, in service of a file most bots never request.
If your actual goal is being cited and recommended by AI systems, the levers that work are elsewhere: entity clarity, third-party corroboration, and extractable content structure. That's generative engine optimisation and answer engine optimisation, and neither of them runs on a text file. If your site has unresolved crawl or indexation problems, start with a technical SEO audit instead.
How to write an llms.txt that's worth having
If you've decided to ship one, do it properly. Fifteen minutes, five steps.
The spec expects a single Markdown # heading with your site or product name, followed by a blockquote giving a short description of what you do. This is the part an agent reads first, so make it unambiguous. Say what you are and who you serve, in plain language.
Something like ## Documentation, ## Guides or ## Company. Keep the sections few and obvious. This is a map, not a taxonomy.
The format is a list item holding the page title, its URL, then a colon and a short description:
- [Page title](/page-url): what this page covers.
The description does the real work. An agent choosing between twelve URLs is choosing based on those descriptions, so write them as answers rather than labels.
## Optional section for the restThe spec reserves this for content an agent can skip when working with a limited context budget. Anything nice-to-have goes here.
Twelve to thirty links is plenty for most sites. A file listing four hundred URLs has defeated its own purpose. Review it quarterly, because a stale llms.txt pointing at deleted pages is worse than no file at all.
Save it as llms.txt in plain UTF-8, upload it to your web root, and confirm it resolves at yourdomain.com/llms.txt. If you're on a documentation platform, check whether it already generates one before you write anything by hand.
The five ways people get this wrong
The single most common implementation error. Enormous maintenance cost, near-zero benefit, and it introduces content drift between the two versions.
Different files, different purposes. One client spent a week conflating them. If your goal is controlling AI training access, that's robots.txt and crawler directives, not this.
It's neither. Keep both. They do jobs llms.txt doesn't touch, and one of them is an actual standard.
Google told you plainly there isn't one. Ship this expecting Search movement and you'll misattribute whatever happens next to the wrong cause.
Every link in that file is a promise. Broken links in an llms.txt say more about your site's maintenance than having no file would.
The numbers, in one place
| Finding | Figure | Source |
|---|---|---|
| Adoption rate across ~300,000 domains | ~10.1% | SE Ranking, 2026 |
| Files never fetched by any bot in May 2026 | 97% | Ahrefs, 2026 |
| Adoption by tier low / mid / high traffic | 9.9% / 10.5% / 8.3% | SE Ranking, 2026 |
| Effect on AI citations modelled two ways | Zero | SE Ranking, 2026 |
| v2 spec released | 10 Aug 2026 | llms.txt spec |
| Google Search impact | None | Google, Jun 2026 |
Frequently asked questions
What is an llms.txt file?An llms.txt file is a Markdown document at your site's root that gives AI systems a curated, prioritised map of your most useful pages. Each entry is a link with a short description. It was proposed by Jeremy Howard in 2024 and reached version 2 in August 2026. It's a community convention, not an official web standard.
Will an llms.txt file help my SEO?No. Google's June 2026 documentation states the file has no effect, positive or negative, on Google Search rankings or AI Overviews. Independent analysis of around 300,000 domains found no correlation between having the file and earning AI citations. Ship it for coding agents if that's your audience, but don't expect Search movement.
Should I create an llms.txt file?Ship one if your site has documentation, an API, or a developer surface, because coding agents genuinely use it. Ship one if it costs you fifteen minutes and no ongoing maintenance. Skip it if creating it would displace technical SEO fixes, content work, or link building, because those produce measurable results and this doesn't.
Where does llms.txt go, root or elsewhere?The root. It should resolve at yourdomain.com/llms.txt, the same way robots.txt does. The spec also permits subpath files for large sites with distinct sections, but for most sites a single root file is correct. Anything else and agents looking for it won't find it.
How do I see the llms.txt file of a website?Type the domain followed by /llms.txt into your browser. If the site has one, it renders as plain text. This is the fastest way to see how companies like Stripe, Vercel and Cloudflare have structured theirs, and copying a good structure is entirely legitimate here.
What's the difference between llms.txt and robots.txt?Robots.txt is exclusionary and tells crawlers where they may not go. It's an officially recognised standard respected across search engines. llms.txt is selective and editorial, pointing AI systems toward your best content. It has no standards body backing and no enforcement mechanism. You need robots.txt. llms.txt is optional.
Do I need an llms.txt generator?Not for a small site. Twelve to thirty links written by hand will beat anything generated, because the descriptions are where the value sits and a generator can't write those well. If you're on Mintlify or GitBook, check your platform settings, because it may already produce one from your existing docs structure.
Does ChatGPT or Claude read llms.txt?No major AI company, including OpenAI, Google, Anthropic, Meta and Mistral, has publicly committed to reading or acting on llms.txt in production systems. GPTBot has been observed fetching the file occasionally, but fetching isn't the same as using it to source, rank or cite content.
If you were about to spend a sprint on llms.txt because a LinkedIn post made you nervous, don't. Spend fifteen minutes on it if your site has docs, then go back to the work that moves numbers. What deserves your attention isn't a file you write about yourself. It's whether AI systems can find, understand and correctly describe your business when someone asks.
Get an AI visibility audit
We run a sample prompt set against your brand across four engines and show you where you stand against the competitors getting named instead of you. No obligation, no sales sequence.
Get My Free Audit → Takes 2 minutes. No credit card. Response within 24 hours.