What Is llms.txt? How to Create One and Does It Work

llms.txt is the file everyone suddenly wants you to add: a short Markdown file at your site root that's supposed to make AI systems find and understand your content faster. It takes under an hour to build, and roughly a quarter of sites that bother already have one live.

What the FOMO posts skip is what happens after you publish it. Google's own documentation states plainly that Search ignores the file for ranking, and Ahrefs' own log data shows most published files never get a single request in a given month.

That doesn't make llms.txt worthless. It makes it narrower than the hype suggests: what the spec actually requires, what an llms.txt generator can and can't do for you, what Google says about it in its own words, and whether building one is worth your hour.

In this guide:

• The file format and the one section the spec actually requires
• How often llms.txt gets read, based on a 137,210-domain study
• Google's exact, dated position on llms.txt and rankings
• The five steps to build one, and what to leave out
• Who should build one, and how to confirm it actually works

What llms.txt is, and the format it follows

llms.txt is a plain Markdown file at a site's root that gives AI systems a curated map of its most important content, instead of raw HTML pages.

The format traces to Jeremy Howard, who co-founded Answer.AI and proposed it in September 2024. It fixes a specific problem: a language model's context window is small and expensive, and a full HTML page burns tokens on navigation, scripts and boilerplate before a model reaches the actual content.

The spec's structure

The spec is built from six parts, and just one of them is required:

1. An H1 with the site or project name. The only required section.
2. A blockquote summary. One or two sentences describing what the site does.
3. Freeform notes. Context a reader (human or model) would need before following any links.
4. H2-delimited link lists. Grouped links to the pages worth reading, each with a short description.
5. An "Optional" section. Secondary links that are useful but skippable under a tight token budget.
6. llms-full.txt, a separate, fuller file (see below).

The H1 is the only required line; everything else is convention. Assembled from the spec's own structure, a minimal compliant file for a documentation site looks like this:

# Acme Docs

> Acme is a project management tool for engineering teams.

## Docs

- [Quickstart](https://acme.dev/docs/quickstart.md): Install and configure Acme in under five minutes.
- [API reference](https://acme.dev/docs/api.md): Every endpoint, with request and response examples.

## Optional

- [Changelog](https://acme.dev/changelog.md): Release notes going back to v1.0.

llms.txt vs robots.txt and sitemap.xml

The llms.txt vs robots.txt question comes up in almost every setup conversation, and sitemap.xml belongs in it too. The short answer: llms.txt duplicates neither.

Each solves a different problem. Robots.txt controls which crawlers may access which paths, and sitemap.xml lists every indexable URL so a search engine can find them.

llms.txt does neither. It curates a small, hand-picked subset of pages a model should read first, in the order they matter.

Download CSV (CC BY 4.0)

llms-full.txt: the fuller export

llms-full.txt is the fuller version: a full Markdown export of the whole documentation site in one file.

It's built for a model that wants everything at once instead of a curated shortlist. It's useful, and it's also where teams get into trouble: Mintlify's own implementation caps its auto-generated llms.txt index at 100,000 characters, splitting page links into separate files once a site grows past that limit.

That cap doesn't apply to llms-full.txt, which is exactly why it can balloon into a single file nobody, human or model, can use in one pass. Treat the size limit as your own design constraint to manage.

A link inside llms.txt is only as useful to a crawler as what actually loads without JavaScript running first. We ran into that exact problem on our own site:

We migrated our own React single-page app to prerendered static HTML for 20 marketing pages because AI crawlers do not execute JavaScript.

The same logic applies to anything you link from llms.txt: if a linked page only renders after a script runs in the browser, an AI crawler following that link gets nothing useful.

None of this replaces the broader work of making content legible to language models; llms.txt is one narrow file inside our llm optimization guide, which covers the structure, content and citation strategy that actually shapes how models represent your site.

Does llms.txt actually get read

llms.txt gets published far more often than it gets read.

Ahrefs' June 2026 study of 137,210 domains found that 28% of them, about 38,360 sites, publish a valid llms.txt file. Of those roughly 38,360 files, 97% received zero requests in the measured month, May 2026.

Download CSV (CC BY 4.0)

Restated as key numbers:

• 1,151 of roughly 38,360 valid files got any request at all in May 2026; the rest, 97% of them, got none.
• 21.7% of requests came from SEO audit tools, more than double the 10.5% that came from AI agents.
• About 19 requests a month is the average for files that get read at all, an estimate built by dividing Ahrefs' own totals.

That's a thin amount of traffic for a file some teams treat as a visibility lever. None of it tells you whether AI systems are visiting or citing your site more broadly, only whether they're fetching this one file.

If your real goal is measuring AI search analytics for your own site, that's a separate, wider question.

Mission Growth's platform tracks AI citations and visibility for customers. That's a different capability than counting hits on one text file.

What Google says about llms.txt, and why Chrome still checks for it

Google's own developer documentation states plainly that maintaining an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

That line comes from Google's AI optimization guide, dated 2026-07-10. It's unconditional: not "probably won't help much," not "limited effect." Search ignores the file, full stop.

So why does Chrome's Lighthouse tool audit for one? Because Lighthouse's "agentic browsing" audit and Google Search's ranking pipeline are two different systems, checking for two different things:

• Chrome Lighthouse. Its "agentic browsing" audit flags a site only when a server error occurs while fetching llms.txt. If the file simply 404s, the audit is marked Not Applicable, because Google's documentation calls providing the file "optional at the moment."
• Google Search. Its ranking pipeline ignores the file entirely, for both visibility and rankings.

They're not contradicting each other; they're answering different questions.

Who else reads llms.txt

Beyond Google and Chrome, no major AI provider has adopted llms.txt as part of its crawler protocol:

Download CSV (CC BY 4.0)

That gap matters for how you think about the file: it has no ranking effect and no confirmed role in any major model's retrieval pipeline yet; it's closer to a convenience file a subset of tools may choose to read.

This is exactly the distinction the geo vs seo distinction post covers in general: SEO optimizes for ranking signals a search engine's algorithm reads, and GEO optimizes for how an AI system selects and cites content.

llms.txt sits in neither camp cleanly. It's infrastructure for a handful of tools that choose to fetch it, not a signal either pipeline is built around.

How to create an llms.txt file, step by step

Building an llms.txt file takes five steps:

1. Decide what belongs in it. Pick the pages a model or agent would actually need first: your quickstart, your API reference, your core product docs. Leave out anything you wouldn't want indexed as your single most important content, because llms.txt is a public, unauthenticated index of what you're calling most important, so list only what you'd stand behind publishing at that root-level path.
2. Draft the Markdown. Follow the six-part structure above: H1 title, blockquote summary, freeform notes if needed, H2-grouped link lists, and an Optional section for secondary links.
3. Host it at the site root. The file goes at /llms.txt, the same level as robots.txt and sitemap.xml, so tools that check for it by convention can find it without extra configuration. Allowing or blocking individual AI crawlers is a robots.txt decision; llms.txt doesn't touch that layer.
4. Validate it. Confirm the file is reachable and that the pages it links to actually render for a crawler as well as a browser (more on this below).
5. Keep it current. A stale llms.txt pointing at deprecated docs or dead links is worse than none; treat it like a changelog that needs an owner.

Several documentation platforms, Mintlify among them, auto-generate an llms.txt index from your existing docs structure rather than asking you to hand-write one. That's a reasonable shortcut for step 2 if your docs are already organized, though it's worth reading through the generated file once before publishing it rather than trusting the default output blind.

Who should actually build one, and how to validate it

Build llms.txt if coding agents, a support chatbot or documentation search are already a real channel for your content.

Skip it if the only goal is Google Search ranking; Google itself says the file doesn't move that number.

Download CSV (CC BY 4.0)

Is llms.txt worth it? For most teams: yes, for a narrow set of tools, cheap to build, and not the lever to reach for if ranking or broad AI citation is the actual goal.

If what you need is AI search optimization services that go beyond one text file, that's a wider engagement than a root-level file can cover.

How to validate it works

Checking that the file exists isn't the same as checking that the pages it links to are actually readable by a crawler.

A link can resolve without erroring and still be useless if the content only renders after a script runs in the browser, because most AI crawlers don't execute JavaScript. Run a spoofed request against a linked page and read the raw response:

curl -A "GPTBot" https://example.com/docs/quickstart.md

If the response body contains your actual content, a crawler can read it. If it comes back mostly empty markup with a script tag doing the real work, that page is invisible to the systems your llms.txt file is pointing them toward.

This is the same failure mode we fixed by prerendering our own marketing pages to static HTML: a link in an llms.txt file is only as useful as what a crawler actually receives when it follows it.

Two places to check before you call it done:

• The free checker. Run our free llms.txt checker against the file itself before you commit to hand-testing every linked page.
• A live example. missiongrowth.io publishes its own llms.txt and llms-full.txt as a curated plain-text knowledge base for AI crawlers, a real, checkable file in production rather than a hypothetical.

llms.txt is real, cheap to build and worth it for the right audience, but it is not an SEO or GEO ranking lever. Google Search ignores the file outright, and 97% of published files got no requests at all in May 2026.

If coding agents or a chatbot already touch your docs, spend the hour building one and validate it with a real crawler user agent. If ranking is the goal, spend that hour somewhere Google actually looks.