AI
llms.txt Explained: The AI Standard for Website Content
July 20269 min read
Photo by Tara Winstead on Pexels
llms.txt is a proposed convention, introduced by Jeremy Howard of Answer.AI in September 2024, for a Markdown file placed at a website's root at /llms.txt. It gives large language models a curated, human-readable map of a site's most important content, addressing the fact that LLM context windows are too small to ingest entire HTML sites. It is a community proposal, not a W3C or IETF standard.
robots.txt tells crawlers which paths they are allowed to access, and sitemap.xml lists every URL for indexing. llms.txt does neither: it is a curated Markdown index of the most valuable content, written for LLMs to read at inference time rather than for crawl control. It complements robots.txt and sitemaps rather than replacing them.
Largely not yet, and this is the standard's biggest open question. Google's John Mueller stated that no AI service has said it uses llms.txt and that server logs show they do not even check for it. An Evil Martians log study in 2026 found only 37 of about 770 llms.txt fetches came from real AI assistants; the rest were SEO bots. No major provider has committed to consuming it.
Create a plain Markdown file with a required H1 title, an optional blockquote summary, and H2 sections containing bulleted Markdown links to your key pages, ideally clean .md versions. Save it at your site root as /llms.txt. Optionally add an llms-full.txt with full inlined content, or generate expanded context files using Answer.AI's llms_txt2ctx tool.
It depends on your goals. Adoption is broad among docs sites such as Anthropic, Stripe, Cloudflare, Vercel, and thousands of Mintlify-hosted docs, and the file is cheap to produce, so many teams ship one as low-risk hygiene. But there is no proven evidence it improves AI retrieval or citations, and no major LLM vendor has committed to using it, so treat it as an optional bet rather than a guaranteed ranking lever.
Photo by Tara Winstead on Pexels
Key Takeaway
llms.txt is a proposed convention, introduced by Jeremy Howard of Answer.AI in September 2024, for a Markdown file at a website's root that gives large language models a curated map of the site's most important content. It addresses LLM context limits and messy HTML. Adoption on documentation sites is wide, but no major AI provider has confirmed using it, and its real value in 2026 remains unproven.
If you have optimized a site for search engines, llms.txt asks you to optimize for a different reader: the language model. Proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024, it is a single Markdown file you place at your site root — example.com/llms.txt — that hands an AI a clean, curated map of what matters on your site.
The idea spread fast among documentation teams and just as fast drew skepticism about whether any AI actually reads it. Both reactions deserve to be taken seriously, so this post covers the spec, the real adoption, and the evidence on whether it does anything — because a file that is trivial to publish is also trivial to publish for no reason.
The spec's own justification is concrete. Language models have context windows too small to ingest an entire website, and converting a real HTML page — navigation, ads, cookie banners, JavaScript — into clean text for a model is, in the spec's words, both difficult and imprecise. llms.txt sidesteps both problems by offering a short, curated, plain-Markdown entry point at a predictable location, written for a model rather than a browser.
The format is deliberately minimal and strictly ordered. The only required element is an H1 with the site or project name. After that you may add a blockquote summary, free-form Markdown, and H2 sections containing bulleted lists of links, each optionally annotated. A section literally named Optional marks links a model can skip when it needs to trim context to fit. Here is the shape:
# Example Project
> A short summary of the project, with the key information
> needed to understand the rest of the file.
## Docs
- [Quick start](https://example.com/docs/quickstart.md): Set up in 5 minutes
- [API reference](https://example.com/docs/api.md): Full endpoint list
## Optional
- [Changelog](https://example.com/changelog.md): Skippable when trimming context| Aspect | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Primary audience | Search and AI crawlers | Search engines | Language models at inference |
| Job | Control what may be accessed | List all indexable URLs | Point to the most useful content |
| Format | Plain-text rules | XML | Human-readable Markdown |
There are two related files people confuse. llms.txt is the short, curated index of links. llms-full.txt is the community convention for a single file with the full page content inlined, for models that can take everything at once. Answer.AI's own tooling also generates expanded context files named llms-ctx.txt and llms-ctx-full.txt from your llms.txt.
Adoption is real and easy to verify — you can open these files in a browser right now.
Here is the uncomfortable part. Google's John Mueller wrote that, as far as he knows, no AI service has said it uses llms.txt, and that server logs show they do not even check for it — he compared it to the long-dead keywords meta tag and called its value purely speculative for now. Independent data backs the doubt: Evil Martians measured two months of traffic in 2026 and found that of roughly 770 fetches of llms.txt and llms-full.txt, only 37 came from named AI assistants such as Perplexity, ChatGPT, and Claude — the rest were SEO crawlers. Their conclusion was that Mueller was basically right.
Treat any claim that llms.txt boosts your AI visibility with suspicion. As of 2026 no major LLM provider has publicly committed to consuming it as a retrieval or ranking signal, and the measured traffic from real assistants is small. It is a proposal with momentum, not a confirmed ranking lever.
So is it worth doing? My take is a qualified yes, framed honestly as a cheap bet rather than a guaranteed win.
I ship one because it costs an afternoon and might pay off if adoption tips the way robots.txt eventually did. But I hold it loosely. The most honest position in 2026 is the spec's own momentum meeting Mueller's log files: plenty of sites publish llms.txt, and almost nothing yet proves the models are reading it. Publish it as an option, watch your own logs, and let evidence — not hype — decide whether you keep investing.