AI Readiness Checker
Check whether GPTBot, ClaudeBot, PerplexityBot and 27 other AI crawlers can read your site, and whether your llms.txt file is spec-complete.
- 30 AI bots checked
- No signup required
- Free forever
- Results in seconds
llms.txt is a plain text file, written in Markdown, that sits at the root of your site and gives large language models a curated map of your most important pages. Jeremy Howard of Answer.AI proposed it in September 2024. It does not block access or grant it. It points AI models at the content you want them to read.
This llms.txt checker does three jobs on one page. It tests your robots.txt against 30 AI crawlers, from GPTBot to PerplexityBot. It validates your llms.txt against the official spec. Then it hands you a copy-paste fix and an AI Readiness score.
How do you use this checker?
Paste a domain and press check. The tool fetches two files, your robots.txt and your llms.txt, then reads them. Nothing gets crawled and no page gets rendered. You see three panels.
Panel one is the AI Readiness score, from zero to ten, with a one-line summary. Panel two is the crawler access matrix. It covers the full AI crawler list, 30 bots sorted into training, search and user-triggered groups, each marked allowed or blocked. Panel three is your llms.txt status, or, if you have none, the llms.txt generator that builds a spec-valid file from a short form.
Does llms.txt actually work? What the data says
Publishing llms.txt will not change your Google rankings today. Here is what the server logs actually show.
- 28%
- Sites with llms.txt live
- Ahrefs, 137,000 domains
- 97%
- Valid files, zero requests
- Ahrefs, May 2026
- 5.3%
- Real AI-bot share of traffic
- Originality.AI
Four independent studies looked for AI crawlers reading these files. Ahrefs pulled server data for 137,000 domains: 28% had an llms.txt live, yet 97% of the valid files logged zero requests in May 2026, bot or human. Of the requests that did land, Originality.AI's breakdown of the same data put real AI bots at 5.3% of traffic, with GPTBot at 4.51% and ClaudeBot at 0.80%. Semrush ran its own test. It added llms.txt to Search Engine Land in March 2025, then watched the logs from mid-August to late October 2025. In that window, Google-Extended, GPTBot, PerplexityBot and ClaudeBot made zero visits.
Adoption numbers swing widely, and the reason matters. Ahrefs measured 28%. SE Ranking measured 10.13% across 300,000 domains. Rankability measured 8.7% across the Tranco top 1,000. The gap comes from the samples. Ahrefs draws from sites that already run its analytics, which skew SEO-aware, while Rankability counts every unreachable domain against the total. Same file, different denominators.
Google is blunt about it. John Mueller called the file "purely speculative" and said no AI system currently uses it. So here is the honest verdict. llms.txt is a low-cost bet, not a ranking signal. Publish one because it is cheap to maintain and keeps your docs tidy. Do not expect it to move search.
Do AI crawlers respect robots.txt?
Not always, and that is the second reason to check. robots.txt runs on the honor system. A crawler reads it and decides whether to obey. In August 2025, Cloudflare documented Perplexity rotating IPs and user-agents to reach sites that had explicitly blocked it, then removed Perplexity from its verified-bot list. And Perplexity's own documentation states that Perplexity-User, its user-triggered fetcher, generally ignores robots.txt rules.
Our access matrix sorts all 30 bots into three groups, because each group answers to a different block. A user-triggered fetcher such as ChatGPT-User grabs one page, once, when a person asks a live question. A search and citation crawler, PerplexityBot or OAI-SearchBot for instance, determines whether AI answers can cite you, so a Disallow there erases you from those answers. A training crawler, GPTBot or ClaudeBot, harvests pages to build future models, and shutting one out leaves your AI-search visibility untouched.
robots.txt cannot express any of that nuance, which is why three new directives arrived in 2025. Microsoft proposed DisallowAITraining, Google proposed Content-Usage, and Cloudflare shipped Content-Signal, each a one-line way to refuse training use while staying visible to search. Cloudflare's managed robots.txt already carries this default across 3.8 million domains. None is an IETF standard yet. This tool is the rare place that shows all three in one view.
How do you block AI crawlers?
One robots.txt block per bot: a User-agent line naming it, then Disallow: /. This checker writes the snippet for you. Tick the bots in the access matrix, copy the output, and paste it under your existing rules, never over them. One trap: Google-Extended and Googlebot are separate tokens, so blocking the training one leaves your Search indexing alone.
How do you write a good llms.txt?
The spec is short. The file needs exactly one H1 with your site or project name. Under it, add a blockquote summary in one or two sentences. Then group your key links under H2 headings, each link written as a Markdown bullet with a title and an optional note. That is the whole format. llms-full.txt, the larger variant some sites publish, is a convention rather than part of the official spec.
If you would rather not hand-write it, the built-in llms.txt generator takes a name, a summary and your sections, then outputs a spec-valid file you can copy or download. It runs in your browser, so your inputs never reach our server.
What this tool cannot tell you
This checker reads intent, not behavior. It parses the static text of your robots.txt and llms.txt and reports what those files say. It cannot promise that a given crawler will obey them, because, as the Perplexity case shows, some do not.
It also does not impersonate bots. The tool never spoofs a GPTBot user-agent to see whether it can slip past your rules. That would be a live-access test with real ethical and legal weight, and a false result either way. Reading the files you already publish is the honest signal, so that is where we stop.
Frequently asked questions about llms.txt and AI crawlers
Want similar results in your business?
Start with a free audit. See where the lift is before you commit.
How it works
- 01
30-minute audit call
We map your funnel against your goal and pull live data from your channels.
- 02
Lift estimate
You get a written estimate of where the lift is, with a 30-day plan to capture it.
- 03
You decide
Run it with us, run it in-house, or shelve it. No commitment from the audit.