llms.txt is a proposed convention, introduced by Jeremy Howard of Answer.AI in September 2024, for a Markdown file placed at a website's root at /llms.txt. It gives large language models a curated, human-readable map of a site's most important content, addressing the fact that LLM context windows are too small to ingest entire HTML sites. It is a community proposal, not a W3C or IETF standard.

How is llms.txt different from robots.txt?

robots.txt tells crawlers which paths they are allowed to access, and sitemap.xml lists every URL for indexing. llms.txt does neither: it is a curated Markdown index of the most valuable content, written for LLMs to read at inference time rather than for crawl control. It complements robots.txt and sitemaps rather than replacing them.

Do AI crawlers actually use llms.txt?

Largely not yet, and this is the standard's biggest open question. Google's John Mueller stated that no AI service has said it uses llms.txt and that server logs show they do not even check for it. An Evil Martians log study in 2026 found only 37 of about 770 llms.txt fetches came from real AI assistants; the rest were SEO bots. No major provider has committed to consuming it.

How do I create an llms.txt file?

Create a plain Markdown file with a required H1 title, an optional blockquote summary, and H2 sections containing bulleted Markdown links to your key pages, ideally clean .md versions. Save it at your site root as /llms.txt. Optionally add an llms-full.txt with full inlined content, or generate expanded context files using Answer.AI's llms_txt2ctx tool.

Is llms.txt worth it in 2026?

It depends on your goals. Adoption is broad among docs sites such as Anthropic, Stripe, Cloudflare, Vercel, and thousands of Mintlify-hosted docs, and the file is cheap to produce, so many teams ship one as low-risk hygiene. But there is no proven evidence it improves AI retrieval or citations, and no major LLM vendor has committed to using it, so treat it as an optional bet rather than a guaranteed ranking lever.

llms.txt Explained: The AI Standard for Website Content

llms.txt is a Markdown file that hands LLMs a curated map of your site. Here is the spec, who has adopted it, and why Google's John Mueller stays skeptical in 2026.

Key Takeaway

llms.txt is a proposed convention, introduced by Jeremy Howard of Answer.AI in September 2024, for a Markdown file at a website's root that gives large language models a curated map of the site's most important content. It addresses LLM context limits and messy HTML. Adoption on documentation sites is wide, but no major AI provider has confirmed using it, and its real value in 2026 remains unproven.

If you have optimized a site for search engines, llms.txt asks you to optimize for a different reader: the language model. Proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024, it is a single Markdown file you place at your site root — example.com/llms.txt — that hands an AI a clean, curated map of what matters on your site.

The idea spread fast among documentation teams and just as fast drew skepticism about whether any AI actually reads it. Both reactions deserve to be taken seriously, so this post covers the spec, the real adoption, and the evidence on whether it does anything — because a file that is trivial to publish is also trivial to publish for no reason.

The problem: context windows and HTML noise

The spec's own justification is concrete. Language models have context windows too small to ingest an entire website, and converting a real HTML page — navigation, ads, cookie banners, JavaScript — into clean text for a model is, in the spec's words, both difficult and imprecise. llms.txt sidesteps both problems by offering a short, curated, plain-Markdown entry point at a predictable location, written for a model rather than a browser.

What goes in an llms.txt file

The format is deliberately minimal and strictly ordered. The only required element is an H1 with the site or project name. After that you may add a blockquote summary, free-form Markdown, and H2 sections containing bulleted lists of links, each optionally annotated. A section literally named Optional marks links a model can skip when it needs to trim context to fit. Here is the shape:

# Example Project

> A short summary of the project, with the key information
> needed to understand the rest of the file.

## Docs

- [Quick start](https://example.com/docs/quickstart.md): Set up in 5 minutes
- [API reference](https://example.com/docs/api.md): Full endpoint list

## Optional

- [Changelog](https://example.com/changelog.md): Skippable when trimming context

llms.txt vs robots.txt vs sitemap.xml

Aspect	robots.txt	sitemap.xml	llms.txt
Primary audience	Search and AI crawlers	Search engines	Language models at inference
Job	Control what may be accessed	List all indexable URLs	Point to the most useful content
Format	Plain-text rules	XML	Human-readable Markdown

There are two related files people confuse. llms.txt is the short, curated index of links. llms-full.txt is the community convention for a single file with the full page content inlined, for models that can take everything at once. Answer.AI's own tooling also generates expanded context files named llms-ctx.txt and llms-ctx-full.txt from your llms.txt.

Who has actually adopted it

Adoption is real and easy to verify — you can open these files in a browser right now.

Anthropic publishes one for its developer docs at platform.claude.com/docs/llms.txt, complete with H2 sections and a link to an llms-full.txt.
Documentation for Stripe, Cloudflare, Vercel, Zapier, Hugging Face, Supabase, and Cursor ships llms.txt files.
Mintlify began auto-generating llms.txt for every documentation site it hosts in November 2024, adding files to thousands of technical docs overnight.

Do AI crawlers actually read it?

Here is the uncomfortable part. Google's John Mueller wrote that, as far as he knows, no AI service has said it uses llms.txt, and that server logs show they do not even check for it — he compared it to the long-dead keywords meta tag and called its value purely speculative for now. Independent data backs the doubt: Evil Martians measured two months of traffic in 2026 and found that of roughly 770 fetches of llms.txt and llms-full.txt, only 37 came from named AI assistants such as Perplexity, ChatGPT, and Claude — the rest were SEO crawlers. Their conclusion was that Mueller was basically right.

Treat any claim that llms.txt boosts your AI visibility with suspicion. As of 2026 no major LLM provider has publicly committed to consuming it as a retrieval or ranking signal, and the measured traffic from real assistants is small. It is a proposal with momentum, not a confirmed ranking lever.

Should you ship one in 2026?

So is it worth doing? My take is a qualified yes, framed honestly as a cheap bet rather than a guaranteed win.

It is low-cost hygiene: if your docs tooling generates one automatically, as Mintlify does, there is no reason to turn it off.
For a hand-maintained site, writing a good llms.txt forces you to articulate your most important pages — useful even if no model ever reads it.
Do not expect measurable AI-referral gains, and do not let it displace the fundamentals: clean HTML, real content, and a normal sitemap still matter far more.

I ship one because it costs an afternoon and might pay off if adoption tips the way robots.txt eventually did. But I hold it loosely. The most honest position in 2026 is the spec's own momentum meeting Mueller's log files: plenty of sites publish llms.txt, and almost nothing yet proves the models are reading it. Publish it as an option, watch your own logs, and let evidence — not hype — decide whether you keep investing.

Sources