What llms.txt is, and whether your site needs one

llms.txt is a proposal for an LLM-friendly site map, not a web standard. OpenAI, Anthropic, and Google publish their own files, but a 137,000-domain study found 97% of third-party files got zero requests. No credible evidence links it to rankings or AI visibility.

Ý chính

• llms.txt is a proposal by Jeremy Howard (Answer.AI), not a web standard. The spec describes itself that way.
• OpenAI, Anthropic, and Google publish llms.txt files for their own docs. That shows the files exist. It does not show their systems read yours.
• The largest published measurement (137,000 domains, Ahrefs, June 2026) found 97% of the llms.txt files in its panel received zero requests in the study month.
• A 12-week server-log study (EZY.ai, 83 sites, April–July 2026) counted single-digit llms.txt fetches from the OpenAI, Anthropic, and Perplexity crawlers that fetched robots.txt thousands of times on the same sites; Googlebot fetched it 67 times against 5,125 robots.txt requests.
• Bottom line: cheap to ship, reasonable for documentation sites agents already use, with no credible evidence of ranking or visibility effects.

Research question

Site owners keep hearing they "need" an llms.txt file. This article works through four questions: what llms.txt is, what problem it tries to solve, what evidence exists that any AI system uses third-party files, and whether a site should add one. Every factual claim below traces to the source ledger in editorial/what-is-llms-txt/research-ledger.md. Claims are marked as fact, interpretation, opinion, or unknown throughout.

Methodology

Eight sources, all fetched on 2026-09-09: the v2 specification, the original 2024 proposal post, Chrome's Lighthouse audit documentation, the live llms.txt files of OpenAI, Anthropic, and Google (verified directly, including one redirect), and two crawler-measurement studies (Ahrefs' independent analysis and EZY.ai's vendor-conducted study with an open dataset). Two vendor pages (Mintlify, GitBook) were fetched and left out of the core argument. Vendor marketing does not count as independent evidence. One cited-elsewhere page (a DigitalApplied post on Google's position) returned 404 when fetched and is excluded entirely. Sample and measurement limits of each study appear where their numbers appear. Throughout: FACT means directly supported by a cited source; INTERPRETATION means analysis built on cited facts; OPINION is labeled editorial judgment; UNKNOWN means no source was found and no assumption is offered.

What llms.txt proposes

In September 2024, Answer.AI's Jeremy Howard proposed that websites add a /llms.txt Markdown file as a curated starting point for language models: brief background, guidance, and links to LLM-friendly pages. The problem statement is narrow. Assembling context from a website today is ambiguous (crawl the whole sitemap, guess at links, ingest source code), and "site authors know best" what matters. The file can sit at the root or any subpath, covering the URLs beneath it, with the most specific file winning where several apply.

The format is specific: an H1 with the site name (the only required section), a blockquote summary, optional detail prose, then H2 "file list" sections of linked pages with descriptions. A companion proposal puts clean Markdown versions of pages at .md URLs, discoverable through standard Link headers. Version 2 of the spec (August 2026) adds this discovery mechanism and the subpath rules.

What llms.txt is not

It has no standards-body status. The specification calls itself "a proposal to standardise," which states an aim. Chrome's Lighthouse documentation calls it "an emerging convention." We found no record of standards-body adoption, and the spec site runs discussion through a GitHub repository and Discord channel, which is how proposals are run.

It does not work like robots.txt. The spec is direct on this: robots.txt states what access is considered acceptable for automated tools; llms.txt is content consumed on demand when an agent needs information. One covers permission, the other offers guidance. It differs from a sitemap too. Sitemaps list every indexable page for search engines, while llms.txt curates a small set of entry points meant to fit in a context window.

No AI lab requires it. The evidence on this point is firm enough to deserve its own section.

Who publishes one, and what that proves

OpenAI, Anthropic, and Google all publish llms.txt files for their own developer documentation, and each was verified live for this article: OpenAI's developer hub serves one (~5.8KB, indexing sub-files per product area), Anthropic's docs URL redirects to a ~73KB file on its developer platform, and Google's Gemini API docs serve one (~32KB, linking Markdown page versions). Mintlify generates the file automatically for hosted docs sites, per its documentation. Chrome's Lighthouse audits for the file under its agentic-browsing checks, while marking a missing file as "Not Applicable" because "providing the file is optional at the moment."

The facts in the previous paragraph prove files exist and tooling recognizes them. They do not prove any AI system reads third-party files, acts on them, or rewards sites for having them. A lab publishing its own map says nothing about whether its crawlers follow yours. That is INTERPRETATION in the weak sense. It is closer to logic: a file existing is not evidence of its use.

Who reads one: the measurements

Two published studies measured actual fetching, and both point the same way.

Ahrefs analyzed server logs and live traffic for 137,000 domains using its analytics products (published June 2026). Twenty-eight percent published an llms.txt file; 97% of those files received zero requests during the study month of May 2026. Of the requests that arrived, 96% came from bots. Named AI bots (training crawlers, retrieval bots, assistants, and agents combined) accounted for 19.5% of fetches; within that group, GPTBot ranked first and Claude-Code second, ahead of every AI search and assistant bot. About 12% of fetches came from the industry studying itself: GEO/AEO tools, file checkers, and researchers. Zero AI-bot requests arrived for files that do not exist. Nothing goes looking. The study also stresses a generous reading of its own numbers: a request proves a file was fetched, not that anything read it or acted on it.

EZY.ai watched 83 sites with llms.txt deployed and server-side logging for twelve weeks (April–July 2026). OpenAI's crawler family fetched robots.txt 3,990 times and llms.txt 7 times on the same sites in the same window. Anthropic: 3,120 versus 9. Perplexity's bot: 775 versus zero. Googlebot: 5,125 versus 67. The single exception was Meta's external agent, which fetched llms.txt slightly more often than robots.txt. It was the only major AI crawler fetching the file at any scale in this panel. Discovery works (median zero days from deployment to first fetch, among the files that were fetched at all); fetching by the large engines beyond Meta is early.

Three cautions come with these numbers. First, fetching differs from reading, reading differs from using, and using differs from benefiting. Each step needs its own evidence, and only the first step has any. Second, both samples skew: Ahrefs toward SEO-aware sites (its own caveat: treat 28% adoption as an upper bound), EZY toward 83 sites from its own small-business panel whose connection history the study does not fully describe. EZY sells AI-visibility tooling, including llms.txt generation, which is a vendor conflict worth knowing. Third, both windows (May and April–July 2026) age, and behavior could change. The direction is consistent across both datasets, but the magnitudes belong to their panels, not the whole web.

Analysis

The picture across both studies is consistent: a format with limited evidence as a discovery mechanism, while working well as documentation tooling. Coding agents are the clearest audience in the available evidence with a plausible reason to use the format: they follow docs links agents already traverse, and agent infrastructure ranks among the larger AI fetch categories. Everything beyond that remains proposal. Vendor framing ("industry standard," "ready for AI crawlers") runs ahead of every measurement, and the audit-and-plugin ecosystem risks manufacturing obligation: SEO tools flagging a missing file imply a requirement no AI lab has stated. That is INTERPRETATION, built on the cited facts above: the file is cheap, the pressure is real, and evidence of payoff is absent.

Should you ship one

The answer depends on your situation, and the conditions matter more than the file. What follows is OPINION: editorial judgment built on the evidence above, not a finding of any study.

For most sites, accurate and accessible content plus conventional web metadata matter more than adding an experimental discovery file. A file the measured crawlers rarely fetch is unlikely to help; a stale one can hurt.

OPINION, labeled as such: for most sites this is a small implementation worth doing only if the maintenance cost is negligible. It is not a strategy.

This site publishes an llms.txt file, inspectable at /llms.txt, as a live format example. That is a verified fact about this site. It is not evidence that the technique works, and no such inference should be drawn. Publishing a file and benefiting from it are different claims, and only the first is established here.

Conclusion

llms.txt is a well-specified proposal solving a real context problem, published by its author, acknowledged by tooling, and adopted by documentation platforms. By the best measurements available, the crawlers measured so far rarely fetch it. Ship one where agents already work your site; do not treat it as visibility strategy. The evidence supports the file's existence far better than its effect.