Mission Growth

AI Readiness Checker

Check whether GPTBot, ClaudeBot, PerplexityBot and 27 other AI crawlers can read your site, and whether your llms.txt file is spec-complete.

  • 30 AI bots checked
  • No signup required
  • Free forever
  • Results in seconds

llms.txt is a plain text file, written in Markdown, that sits at the root of your site and gives large language models a curated map of your most important pages. Jeremy Howard of Answer.AI proposed it in September 2024. It does not block access or grant it. It points AI models at the content you want them to read.

This llms.txt checker does three jobs on one page. It tests your robots.txt against 30 AI crawlers, from GPTBot to PerplexityBot. It validates your llms.txt against the official spec. Then it hands you a copy-paste fix and an AI Readiness score.

TrainingSearch & citationUser-triggered
30 AI crawlers, each marked allowed or blocked by your robots.txt
How it works

How do you use this checker?

Paste a domain and press check. The tool fetches two files, your robots.txt and your llms.txt, then reads them. Nothing gets crawled and no page gets rendered. You see three panels.

Panel one is the AI Readiness score, from zero to ten, with a one-line summary. Panel two is the crawler access matrix. It covers the full AI crawler list, 30 bots sorted into training, search and user-triggered groups, each marked allowed or blocked. Panel three is your llms.txt status, or, if you have none, the llms.txt generator that builds a spec-valid file from a short form.

The evidence

Does llms.txt actually work? What the data says

Publishing llms.txt will not change your Google rankings today. Here is what the server logs actually show.

28%
Sites with llms.txt live
Ahrefs, 137,000 domains
97%
Valid files, zero requests
Ahrefs, May 2026
5.3%
Real AI-bot share of traffic
Originality.AI

Four independent studies looked for AI crawlers reading these files. Ahrefs pulled server data for 137,000 domains: 28% had an llms.txt live, yet 97% of the valid files logged zero requests in May 2026, bot or human. Of the requests that did land, Originality.AI's breakdown of the same data put real AI bots at 5.3% of traffic, with GPTBot at 4.51% and ClaudeBot at 0.80%. Semrush ran its own test. It added llms.txt to Search Engine Land in March 2025, then watched the logs from mid-August to late October 2025. In that window, Google-Extended, GPTBot, PerplexityBot and ClaudeBot made zero visits.

Adoption numbers swing widely, and the reason matters. Ahrefs measured 28%. SE Ranking measured 10.13% across 300,000 domains. Rankability measured 8.7% across the Tranco top 1,000. The gap comes from the samples. Ahrefs draws from sites that already run its analytics, which skew SEO-aware, while Rankability counts every unreachable domain against the total. Same file, different denominators.

Google is blunt about it. John Mueller called the file "purely speculative" and said no AI system currently uses it. So here is the honest verdict. llms.txt is a low-cost bet, not a ranking signal. Publish one because it is cheap to maintain and keeps your docs tidy. Do not expect it to move search.

Do AI crawlers respect robots.txt?

Not always, and that is the second reason to check. robots.txt runs on the honor system. A crawler reads it and decides whether to obey. In August 2025, Cloudflare documented Perplexity rotating IPs and user-agents to reach sites that had explicitly blocked it, then removed Perplexity from its verified-bot list. And Perplexity's own documentation states that Perplexity-User, its user-triggered fetcher, generally ignores robots.txt rules.

Our access matrix sorts all 30 bots into three groups, because each group answers to a different block. A user-triggered fetcher such as ChatGPT-User grabs one page, once, when a person asks a live question. A search and citation crawler, PerplexityBot or OAI-SearchBot for instance, determines whether AI answers can cite you, so a Disallow there erases you from those answers. A training crawler, GPTBot or ClaudeBot, harvests pages to build future models, and shutting one out leaves your AI-search visibility untouched.

robots.txt cannot express any of that nuance, which is why three new directives arrived in 2025. Microsoft proposed DisallowAITraining, Google proposed Content-Usage, and Cloudflare shipped Content-Signal, each a one-line way to refuse training use while staying visible to search. Cloudflare's managed robots.txt already carries this default across 3.8 million domains. None is an IETF standard yet. This tool is the rare place that shows all three in one view.

How do you block AI crawlers?

One robots.txt block per bot: a User-agent line naming it, then Disallow: /. This checker writes the snippet for you. Tick the bots in the access matrix, copy the output, and paste it under your existing rules, never over them. One trap: Google-Extended and Googlebot are separate tokens, so blocking the training one leaves your Search indexing alone.

TrainingGPTBotClaudeBotSearch & citationOAI-SearchBotPerplexityBotUser-triggeredChatGPT-UserYour siterobots.txt · llms.txt
The three crawler groups that reach your site, and the block each one answers to

How do you write a good llms.txt?

The spec is short. The file needs exactly one H1 with your site or project name. Under it, add a blockquote summary in one or two sentences. Then group your key links under H2 headings, each link written as a Markdown bullet with a title and an optional note. That is the whole format. llms-full.txt, the larger variant some sites publish, is a convention rather than part of the official spec.

If you would rather not hand-write it, the built-in llms.txt generator takes a name, a summary and your sections, then outputs a spec-valid file you can copy or download. It runs in your browser, so your inputs never reach our server.

/llms.txtMarkdown
#Your project name>One or two sentences on what the site is about.##Docs-[Quickstart](/docs/quickstart): setup guide-[API reference](/docs/api)##Guides-[Best practices](/guides/best-practices)
The whole spec: one H1, a blockquote summary, links under H2 headings

What this tool cannot tell you

This checker reads intent, not behavior. It parses the static text of your robots.txt and llms.txt and reports what those files say. It cannot promise that a given crawler will obey them, because, as the Perplexity case shows, some do not.

It also does not impersonate bots. The tool never spoofs a GPTBot user-agent to see whether it can slip past your rules. That would be a live-access test with real ethical and legal weight, and a false result either way. Reading the files you already publish is the honest signal, so that is where we stop.

Frequently asked questions about llms.txt and AI crawlers

Does Google support llms.txt?

No. Google's John Mueller compared llms.txt to the old keywords meta tag and called it "purely speculative". Google's June 2026 update to its generative-AI guidance put the position in writing: this kind of file has no effect on Search rankings or AI Overviews.

Is llms.txt worth it in 2026?

It is a low-cost bet. No independent server-log study has yet found meaningful AI-crawler traffic to these files, and Ahrefs saw 97% of valid llms.txt files get zero requests in May 2026. Publish one because it is cheap to maintain and keeps your documentation tidy, not because it will lift rankings.

How do I block GPTBot?

Add two lines to your robots.txt: User-agent: GPTBot then Disallow: /. OpenAI documents that GPTBot honors this, and the change takes about a day to register. Blocking GPTBot stops training collection only. It does not remove you from ChatGPT search, which runs on a separate crawler called OAI-SearchBot.

What is the difference between GPTBot and OAI-SearchBot?

They are two OpenAI crawlers with different jobs. GPTBot gathers data to train foundation models. OAI-SearchBot powers ChatGPT's search feature and decides whether your pages can appear as citations. Blocking GPTBot protects your content from training while keeping you eligible for ChatGPT search. Blocking OAI-SearchBot does the reverse.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they may not fetch. llms.txt does the opposite. It points AI models toward the pages you most want read, in clean Markdown. robots.txt is an access rule that predates AI, and llms.txt is a content map with no enforcement power. You can, and often should, publish both.

Do AI bots actually read llms.txt files?

Rarely, so far. Semrush put llms.txt on Search Engine Land in March 2025, and between mid-August and late October 2025 the logs showed zero visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot. John Mueller of Google said in 2025 that no AI system currently uses the file. Adoption is climbing, but crawler pickup has not followed.

What is llms-full.txt, and is it part of the spec?

llms-full.txt is a larger file that inlines your full documentation in one place, rather than linking to it. It is a popular convention, but the official llmstxt.org spec does not define it. Originality.AI tracked it growing from 23 instances to 2,463 over a year, so adoption is real even without formal status.

What is ai.txt, and how is it different from llms.txt?

ai.txt is a separate format that signals AI-usage permissions by file type. It is not llms.txt and solves a different problem. Originality.AI tracks it as its own file, growing from 4 to 397 instances over a year. llms.txt maps your content for models to read. This checker validates llms.txt, not ai.txt, so keep the two apart.
Next step

Want similar results in your business?

Start with a free audit. See where the lift is before you commit.

How it works

  1. 01

    30-minute audit call

    We map your funnel against your goal and pull live data from your channels.

  2. 02

    Lift estimate

    You get a written estimate of where the lift is, with a 30-day plan to capture it.

  3. 03

    You decide

    Run it with us, run it in-house, or shelve it. No commitment from the audit.