llms.txt is a proposed file that tells AI systems what your site contains. It costs an hour to write and it is widely recommended. It is also widely oversold — no major AI provider has committed to reading it.
Written by PX7 Digital, who publish llms.txt on their own sites and measure what AI answers actually cite.
A plain file at /llms.txt describing what the site is, what it offers, and which pages matter — written for a language model rather than a browser. The proposal was published in 2024 by Jeremy Howard, and the format is deliberately simple: a heading, a summary line, and lists of links with one-sentence descriptions.
robots.txt is a decades-old standard that crawlers genuinely obey. llms.txt borrows the placement and the plain-text simplicity, but it is a proposal, not a standard, and obedience is the open question rather than the premise.
It does not block anything or grant permission. It says here is what we are and where the good pages are. Access control still belongs in robots.txt and in your provider-specific crawler settings.
OpenAI, Anthropic, Google and Perplexity have not committed to using llms.txt. Google has publicly said it is not used for Search. Anyone telling you it drives AI citations is describing a hope, not a documented behaviour.
In the answer evidence we collect, assistants name conventional sources — app stores, code hosts, established review sites, documentation and the pages themselves. We have not seen an assistant attribute an answer to a site's llms.txt.
An hour of work, no ongoing cost, no risk of penalty, and a real chance the convention gains adoption. Writing one is a reasonable bet. Expecting it to change your answers this quarter is not.
Check that GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Googlebot receive your actual pages and not a JavaScript shell. A site whose content only exists after scripts run is invisible to much of the ecosystem, whatever its llms.txt says.
Prices, versions, requirements and claims that agree between your visible copy and your structured data. Assistants that cannot confirm a fact tend to omit it — or invent a plausible substitute.
App stores, package registries, code hosts and independent write-ups. This is slower and harder than adding a file, which is exactly why the file is more popular advice.
Assistants summarise pages that address the user's actual question. A page comparing the real options in a category earns citation in a way a product page rarely does.
A stale llms.txt is worse than none: it hands a confident summary of last year's product to anything that does read it. Update it in the same commit as the change it describes.
One line per page, saying what the page answers. The file is a map, not a pitch — and a model reading a pitch has nothing to summarise.
Content negotiation on Accept: text/markdown, or a plain index.md beside each page, gives any agent the text without the markup. It costs about as little as llms.txt and helps every reader that arrives.
AICanary checks the technical foundation for free, then runs real customer questions against the major assistants to see whether you appear, who is recommended instead, and which sources shape the answer.
Run the free report