What llms.txt is, and whether your site needs one

llms.txt is a proposal for an LLM-friendly site map, not a web standard. OpenAI, Anthropic, and Google publish their own files, but a 137,000-domain study found 97% of third-party files got zero requests. No credible evidence links it to rankings or AI visibility.

Key takeaways

• llms.txt is a proposal by Jeremy Howard (Answer.AI), not a web standard. The spec says so itself.
• OpenAI, Anthropic, and Google all publish llms.txt files for their own docs. That proves the files exist — it says nothing about whether their systems read yours.
• The largest published measurement (137,000 domains, Ahrefs, June 2026) found 97% of the llms.txt files in its panel received zero requests in the study month.
• A 12-week server-log study (EZY.ai, 83 sites, April–July 2026) counted single-digit llms.txt fetches from the OpenAI, Anthropic, and Perplexity crawlers that fetched robots.txt thousands of times on the same sites; Googlebot fetched it 67 times against 5,125 robots.txt requests.
• Verdict in brief: cheap to ship, reasonable for documentation sites agents already use, with no credible evidence of ranking or visibility effects.

Research question

Site owners keep hearing that they "need" an llms.txt file. This article asks four questions: what is llms.txt, what problem is it intended to solve, what evidence exists that any AI system actually uses third-party files, and should a website implement one? Every factual claim below traces to the source ledger in editorial/what-is-llms-txt/research-ledger.md; claims are separated into fact, interpretation, opinion, and unknown throughout.

Methodology

Eight sources, all fetched on 2026-09-09: the v2 specification, the original 2024 proposal post, Chrome's Lighthouse audit documentation, the live llms.txt files of OpenAI, Anthropic, and Google (verified directly, including one redirect), and two crawler-measurement studies (Ahrefs' independent analysis and EZY.ai's vendor-conducted study with an open dataset). Two vendor pages (Mintlify, GitBook) were fetched but deliberately excluded from the core argument — vendor marketing is not independent evidence. One cited-elsewhere page (a DigitalApplied post on Google's position) returned 404 when fetched and is excluded entirely. Sample and measurement limits of each study are stated where their numbers appear. Throughout: FACT means directly supported by a cited source; INTERPRETATION means analysis built on cited facts; OPINION is labeled editorial judgment; UNKNOWN means no source was found and no assumption is offered.

What llms.txt proposes

In September 2024, Answer.AI's Jeremy Howard proposed that websites add a /llms.txt Markdown file giving language models a curated starting point: brief background, guidance, and links to LLM-friendly pages. The problem it targets is real and narrowly stated — assembling context from a website today is ambiguous (crawl the whole sitemap? guess at links? ingest source code?), and "site authors know best" what matters. The file can sit at the root or any subpath, covering the URLs beneath it, with the most specific file winning where several apply.

The format is precise: an H1 with the site name (the only required section), a blockquote summary, optional detail prose, then H2 "file list" sections of linked pages with descriptions. A companion proposal puts clean Markdown versions of pages at .md URLs, discoverable through standard Link headers. Version 2 of the spec (August 2026) adds this discovery mechanism and the subpath rules.

What llms.txt is not

It is not a standard. The specification calls itself "a proposal to standardise" — an aspiration, not a status. Chrome's Lighthouse documentation calls it "an emerging convention." We found no record of standards-body adoption, and the spec site hosts discussion through a GitHub repository and Discord channel, which is how proposals, not standards, are governed.

It is not robots.txt. The spec is explicit: robots.txt tells automated tools what access is considered acceptable; llms.txt is content consumed on demand when an agent needs information. One governs permission, the other offers guidance. It is not a sitemap either — sitemaps enumerate every indexable page for search engines, while llms.txt curates a small set of entry points meant to fit in a context window.

And it is not a requirement from any AI lab. This is the claim the evidence most firmly refuses to support, so it gets its own section.

Who publishes one — and what that proves

All three — OpenAI, Anthropic, and Google — publish llms.txt files for their own developer documentation, and each was verified live for this article: OpenAI's developer hub serves one (~5.8KB, indexing sub-files per product area), Anthropic's docs URL redirects to a ~73KB file on its developer platform, and Google's Gemini API docs serve one (~32KB, linking Markdown page versions). Mintlify generates the file automatically for hosted docs sites, per its documentation. Chrome's Lighthouse even audits for the file's presence under its agentic-browsing checks — while explicitly marking a missing file as "Not Applicable" because "providing the file is optional at the moment."

Here is the distinction the article must not blur: every fact in the previous paragraph proves that files exist and tooling acknowledges them. None of it proves that any AI system reads third-party files, acts on them, or rewards sites for having them. A lab publishing its own map says nothing about whether its crawlers follow yours. That is INTERPRETATION only in the weak sense — it is closer to logic: existence of a file is not evidence of its consumption.

Who reads one — the measurements

Two published studies measured actual fetching, and both point the same direction.

Ahrefs analyzed server logs and live traffic for 137,000 domains using its analytics products (published June 2026). Twenty-eight percent published an llms.txt file; 97% of those files received zero requests during the study month of May 2026. Of the requests that did arrive, 96% came from bots. Named AI bots — training crawlers, retrieval bots, assistants, and agents combined — accounted for 19.5% of fetches; within that group, GPTBot ranked first and Claude-Code second, ahead of every AI search and assistant bot. About 12% of fetches came from the industry studying itself: GEO/AEO tools, file checkers, and researchers. Zero AI-bot requests arrived for files that don't exist — nothing goes looking. And the study stresses the generous reading of its own numbers: a request proves a file was fetched, not that anything read it or acted on it.

EZY.ai watched 83 sites with llms.txt deployed and server-side logging for twelve weeks (April–July 2026). OpenAI's crawler family fetched robots.txt 3,990 times and llms.txt 7 times on the same sites in the same window. Anthropic: 3,120 versus 9. Perplexity's bot: 775 versus zero. Googlebot: 5,125 versus 67. The single exception was Meta's external agent, which fetched llms.txt slightly more often than robots.txt — the only major AI crawler fetching the file at any scale in this panel. Discovery itself works fine (median zero days from deployment to first fetch, among the files that were fetched at all); fetching by the big engines beyond Meta is what's early.

Three cautions travel with these numbers. First, fetching is not reading, reading is not using, and using is not benefiting — each step needs its own evidence, and only the first step has any. Second, both samples skew: Ahrefs toward SEO-aware sites (its own caveat: treat 28% adoption as an upper bound), EZY toward 83 sites from its own small-business panel whose connection history the study doesn't fully describe — and EZY sells AI-visibility tooling, including llms.txt generation, which is a vendor conflict worth knowing. Third, both windows (May and April–July 2026) age, and behavior could change. The direction is consistent across both datasets, but the magnitudes belong to their panels, not the whole web.

Analysis

The consistent picture is a format showing limited evidence of adoption as a discovery mechanism, while working well as documentation tooling. Coding agents are the clearest audience identified in the available evidence with a plausible reason to use the format — they follow docs links agents already traverse, and agent infrastructure ranks among the larger AI fetch categories. Everything beyond that is proposal. The vendor framing ("industry standard," "ready for AI crawlers") runs ahead of every measurement, and the audit-and-plugin ecosystem risks manufacturing obligation — SEO tools flagging a missing file imply a requirement that no AI lab has stated. That is INTERPRETATION, built on the cited facts above: the file is cheap, the pressure is real, and the evidence of payoff is absent.

Should you ship one

The honest answer is conditional, and the conditions matter more than the file. What follows is OPINION — editorial judgment built on the evidence above, not a finding of any study.

For most sites, maintaining accurate, accessible content and conventional web metadata remains more fundamental than adding an experimental discovery file. A file the measured crawlers rarely fetch is unlikely to help you; a stale one can hurt.

OPINION, labeled as such: for most sites this is a small implementation worth doing only if the maintenance cost is negligible — not a strategy.

This site publishes an llms.txt file, inspectable at /llms.txt, as a live format example. That is a verified fact about this site. It is not evidence that the technique works, and no such inference should be drawn — publishing a file and benefiting from it are different claims, and only the first is established here.

Conclusion

llms.txt is a well-specified proposal solving a real context problem, published by its author, acknowledged by tooling, adopted by documentation platforms — and, by the best measurements available, rarely fetched by the crawlers measured so far. Ship one where agents already work your site; don't mistake it for visibility strategy. The evidence supports the file's existence far better than its effect, and an honest publication should say exactly that.