llms.txt, GPTBot, ClaudeBot: A Small Site's Guide to AI Crawler Hygiene

If you run a small site, there is a decent chance your robots.txt hasn't been touched since you added a single GPTBot rule in 2023 — or that you've never touched it at all. That gap matters more than it used to. By mid-2025, Cloudflare's network data showed training-related crawling had grown to roughly 80% of all AI bot activity, up from 72% a year earlier.^[3] AI crawlers are now a meaningful slice of who knocks on your server's door every day, and the rules you give them decide two very different things: whether your content trains someone else's model for free, and whether you show up when a buyer asks ChatGPT or Perplexity a question.

The thing that makes 2026 genuinely different is that the major AI vendors split their single crawler into several. OpenAI no longer runs one bot — it runs GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for on-demand fetches. Anthropic runs three. The practical consequence: the instinctive move of Disallow: / for every AI user-agent now does two jobs at once. It opts you out of training corpora (often what you wanted) and deletes you from AI search answers (almost never what you wanted). Digital Applied's analysis of Anthropic's framework reports that roughly 71% of top news publishers block at least one retrieval or search bot, frequently while intending to block only training.^[5] That is the exact, expensive mistake this guide is built to prevent.

llms.txt is a community-proposed Markdown file you place at your domain root (/llms.txt) that lists your most important pages in clean, parseable form, so a large language model can find and understand your best content without wading through navigation, ads, and scripts. It was proposed by Jeremy Howard of Answer.AI in September 2024. The analogy people reach for is "a sitemap for LLMs," and the intent is genuinely sensible: give the models a curated, low-noise map of what matters on your site, optionally with a fuller /llms-full.txt that inlines the actual content.

The problem is the gap between the idea and the evidence. After eighteen months of industry conversation, the data is sobering:

• Adoption is about one in ten sites. SE Ranking's study of 300,000 domains found a 10.13% adoption rate, and crucially, adoption was nearly identical across low-, mid-, and high-traffic tiers (~9–10% each) — so it isn't the sophisticated sites racing ahead.^[1]
• No measurable citation lift. The same SE Ranking analysis found no statistically significant difference in AI citation frequency between sites with and without an llms.txt. A model trained on AI-citation data actually improved when the llms.txt variable was removed.^[1]
• Almost nothing reads it. Limy.AI monitored over 500 million AI bot events across 90 days and found only 408 requests that targeted llms.txt directly.^[14] Search Engine Land tracked 10 sites for 90 days before and after adding the file; only two saw AI traffic increases, and not because of the file.^[2]
• No official support. As of mid-2026, none of OpenAI, Anthropic, Google, or Perplexity has officially confirmed that its systems read or act on llms.txt. It remains a community spec, not an adopted standard.

The takeaway is not "llms.txt is useless" — it's that the two files are not interchangeable, and the one that actually changes outcomes today is the boring, decades-old one. If you only have time to get one file right in 2026, make it robots.txt, with rules that reflect the current AI crawler landscape rather than the 2023 version of it.

Before you can write a sensible rule, you need to know what each bot is for. Every AI crawler does one of three jobs: it collects pages for model training, it indexes pages for AI search answers, or it fetches a page in real time because a user asked the assistant about it right now. These are different commercial relationships, and in 2026 the major vendors finally expose them as different bots you can control independently.

Two rows deserve a second read. Google-Extended is not a real crawler at all — it's a control token that never appears in your server logs as an HTTP request. It only governs whether Google may use pages it has already crawled (via the normal Googlebot) to train Gemini and Vertex AI. And Bytespider (ByteDance) has a documented history of ignoring robots.txt inconsistently, which is why blocking it requires a firewall or IP-level rule rather than a polite Disallow.

The business argument for blocking training crawlers comes down to a single ratio: how many of your pages a bot crawls for every one visitor it sends back. Cloudflare publishes this crawl-to-referral ratio across its network, and the spread between vendors is extraordinary. Traditional Googlebot sits around 5 pages crawled per referral. Anthropic's ClaudeBot, at its June 2025 peak, was crawling roughly 70,900 pages for every visitor it referred back — an asymmetry that reframes training access as a one-way extraction of value.^[5]

But there's a catch that stops "just block the training bots" from being a free lunch, and it's the most important nuance in this whole article. Blocking has citation side-effects. AuthorityTech's 2026 analysis found that sites blocking Google-Extended are significantly less likely to be cited by generative engines — even in AI Overviews, where Google technically retains content access through the regular index.^[11] And ppc.land's data shows blocking is leaky in the other direction too: blocking AI crawlers doesn't reliably stop citations, because engines pull from alternate paths, third-party quotes, and cached copies.^[10] You can lose the visibility upside without fully gaining the privacy you blocked for.

On the other side of the ledger sits the traffic damage that pushes people to block in the first place. Google AI Overviews have measurably cut referral traffic: Digital Content Next reports 1–25% traffic declines for members, averaging around 25%, and click-through rates drop anywhere from 34.5% to 79% when an AI Overview appears, depending on query type.^[8] Publishers surveyed by ppc.land expect a further 43% traffic decline over the next three years.^[10] When AI is both taking your traffic and crawling you 70,000-to-1, the urge to pull up the drawbridge is understandable.

For two years, the single most-asked question about AI crawlers was some version of: "How do I appear in normal Google Search but not in AI Overviews?" Until June 2026, the honest answer was "you can't." Google-Extended — the token most people reached for — only controls whether Google trains Gemini and Vertex AI on your already-crawled pages. It has never controlled AI Overviews or AI Mode, which draw on the standard Googlebot index. Blocking Google-Extended did nothing to keep you out of the AI summaries actually eating your clicks.^[8]

That changed on June 3, 2026, when Google shipped a Search Console AI performance report alongside an opt-out toggle that lets publishers remove content from AI Overviews and AI Mode without losing standard Search ranking.^[6] The setting takes effect June 17, 2026, when Google begins acting on the signal, and activating it does not affect your ranking in regular Google Search results.^[7] Two caveats worth reading twice: the standalone Gemini app is excluded from this opt-out (it's a separate product), and opting out means accepting whatever citation/visibility you were getting from AIO disappears too.

There is no single correct configuration — there's a correct configuration for your goal. Here are the three profiles that cover almost every small site, from "I want to be in every answer" to "keep me out of AI entirely."

For most small sites, the honest recommendation is Profile 1 — allow everything. Your goal is visibility, the bandwidth cost of AI crawling is trivial at small scale, and blocking training crawlers carries citation side-effects you can't fully predict. The "block training, keep citations" profile (Profile 2) is the right default for content businesses, publishers, and anyone shipping original research who genuinely resents one-way extraction. Profile 3 is for paywalled, membership, or legally-sensitive content only.

Rules written in the dark are guesses. Spend twenty minutes looking at who's actually crawling you before you change a single line, because the bot landscape varies wildly by niche. Here's the practical monitoring stack, cheapest first:

• Server / CDN access logs. Filter by user-agent for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, and Bytespider. This tells you frequency and which pages each bot hits. Note: Google-Extended will never appear — it's a token, not a request.
• Cloudflare Radar AI Insights. If you're behind Cloudflare, the free AI Insights dashboard shows AI crawler activity and the crawl-to-referral data referenced throughout this article.
• Google Search Console. The new AI performance report (June 2026) is where you'll see AI Overviews / AI Mode impressions — and where the opt-out toggle lives.
• Verify authenticity. Spoofed user-agents are common. Cross-check suspicious hits against published IP ranges — OpenAI lists them at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json.

→ Do this now: Open https://your-domain.com/robots.txt in a browser. If it doesn't mention OAI-SearchBot or Claude-SearchBot, it's out of date — and you may be invisible to the exact AI search products your customers are using. Pick a profile from the table above, paste the matching config, and you've done more crawler hygiene than the overwhelming majority of small sites have in 2026.