website menu

Is llms.txt the “Website Manual” for the AI Era

If you’ve been following technical SEO or AI crawler optimization lately, you’ve probably seen one filename come up again and again: llms.txt. Supporters call it “a sitemap for AI,” while skeptics dismiss it as useless. Chrome Lighthouse has even added it as an audit item. So what exactly is this file, and in 2026, does it actually do anything?

What Is llms.txt?

Put simply, llms.txt is a Markdown file placed in your site’s root directory. Its core purpose isn’t to control crawler access like robots.txt does—it’s to provide large language models and AI agents with a curated, clean index of your content.

Imagine an AI coding assistant trying to answer a question about a library’s API. Instead of struggling through HTML pages cluttered with navigation bars, ads, and JavaScript, it could read llms.txt and immediately find links to clean Markdown versions of the docs. That significantly reduces token consumption and improves answer accuracy.

The proposal was first introduced by Jeremy Howard in September 2024 and updated to v2 in August 2026. A key change in v2 was the introduction of standard link relations (such as rel="describedby"), allowing AI agents to more automatically discover llms.txt and the Markdown pages it points to.

How Do You Write a Standard llms.txt?

The format is very lightweight—essentially structured Markdown. A compliant llms.txt typically includes the following:

  1. An H1 heading: The name of the project or website (this is the only required element).
  2. A blockquote summary: One sentence describing what the site does.
  3. Sectioned link lists: H2 headings for categories (e.g., ## Docs, ## Pricing), with lists of linked core pages and optional short descriptions.

A typical example looks something like this:

markdown

# NameSilo

> Domain registration and web hosting since 2009.

## Docs
- [Domain Registration Guide](https://www.namesilo.com/support/v2/articles/domain-registration)
- [DNS Management](https://www.namesilo.com/support/v2/articles/dns-management)

## Pricing
- [Domain Prices](https://www.namesilo.com/pricing)
- [Hosting Plans](https://www.namesilo.com/hosting)

## Policies
- [Terms of Service](https://www.namesilo.com/terms)
- [Privacy Policy](https://www.namesilo.com/privacy)

For content-heavy sites, you can also generate an llms-full.txt that concatenates the full text of all linked pages into a single file, letting an AI ingest the entire knowledge base in one go.

The Controversy and the Reality: Is Anyone Actually Reading It?

This is the core question. Despite the elegant format, no major AI provider has publicly stated that it actively crawls or uses llms.txt.

Google’s John Mueller has publicly said that as far as he knows, “no AI system is currently using llms.txt,” comparing it to the long-abandoned keywords meta tag—a site unilaterally declaring its own importance that search engines simply ignore. A third-party study from 2025 (covering nearly 300,000 domains) supports this: while roughly 10% of sites had deployed the file, fewer than 1% of AI citations involved an llms.txt, with no meaningful correlation between the two.

That said, it isn’t entirely useless. The reality is polarized:

  • In developer documentation, it’s a genuine need: Sites like Anthropic, Stripe, Cloudflare, and the many docs built on Mintlify already support llms.txt natively. For AI coding tools like Cursor and Claude Code, reading llms.txt is already standard practice for gathering context.
  • For ordinary blogs or e-commerce sites, it may be a placebo: If you’re hoping that dropping in an llms.txt will make ChatGPT prioritize recommending you, you’ll likely be disappointed. Google’s Lighthouse audit even states explicitly that if the file returns a 404, the audit is marked “not applicable” (N/A)—because providing it is optional.

Should You Deploy It?

Yes, deploy it—but don’t harbor unrealistic expectations.

If your site contains technical documentation, API references, or product manuals, or if you want AI coding assistants to read your open-source project more smoothly, then spending 10 minutes writing an llms.txt is worth it. Maintenance costs are minimal, and there’s no downside. You can write it by hand or generate it automatically with tools like llms-generator.

But if you run an ordinary content site or e-commerce store, deprioritize it. Rather than obsessing over llms.txt, make sure your robots.txt doesn’t accidentally block GPTBot or ClaudeBot, and ensure your core pages have clean HTML structure and load quickly. After all, as Mueller said, AI will ultimately look at your website’s content directly—not just take your word for it.

In the end, llms.txt is an elegant proposal that precisely captures the friction between AI agents and the existing web architecture. But as of 2026, it’s more of a bet on the future than a switch that will immediately bring you traffic.