The 2026 Bot Map: Who Is Actually Crawling Your Site, and What You Can Safely Block

Blocking “AI bots” as a single category is the expensive mistake of 2026. Two entirely different families hide under that label: training crawlers, which pull your content to train a model and return nothing, and answer-time fetchers, which are dispatched in real time to answer a user’s question — and cite you by name in the response. They ship distinct user-agents, so three lines in robots.txt are enough to separate them. According to Cloudflare’s report of 1 July 2026, 52% of crawler requests are now for AI training as of June 2026, up from 22% in spring 2025, and for the first time more than 50% of traffic on the internet is non-human. Block everything and you erase yourself from the answers; block nothing and you pay bandwidth for training that never comes back to you.

Two families, not one

The distinction that decides everything is not who operates the bot but when it runs. A training crawler works on its own cycles, unconnected to any user. It collects text for a corpus, and your content dissolves into the model weights with no source, no credit and no link. An answer-time fetcher behaves completely differently: a user asked a question, the system decided your page is relevant, and the request goes out now to read it and quote from it — with your brand name and a link back.

Three companies have explicitly split the two families into separate user-agents, and that is precisely the point most sites miss. OpenAI runs GPTBot for training and OAI-SearchBot for indexing in ChatGPT Search. Anthropic runs ClaudeBot for training and Claude-SearchBot for indexing. Google runs Google-Extended as an opt-out control for Gemini training and grounding — a control that is not a crawler in its own right and has no effect on Search ranking.

There is a third, less familiar layer: the real-time fetch initiated by a person. When someone pastes a link into ChatGPT and asks for a summary, the request comes from ChatGPT-User, not GPTBot. At Anthropic it is Claude-User; at Perplexity, Perplexity-User. These bots neither build an index nor train anything — they serve a human who already asked for you.

The table: who’s who, and what blocking costs

Googlebot is the case you cannot split

Google does not separate the crawling that feeds ordinary search results from the crawling that feeds AI Overviews and AI Mode. Both are Googlebot, and both draw on the same index. Blocking Googlebot in order to “stay out of AI” means leaving Google altogether. Google-Extended governs only the separate Gemini training and grounding layer, and does not remove you from Search.

The practical conclusion is simple: as far as Google is concerned, the decision has already been made for you. The only question left in your hands is how your page reads when a system generates an answer from it — and that is a question of structure and substance, not permissions. I go into it in the post on measuring visibility in AI answer engines.

Cloudflare adds one figure that explains why classification is hard in the first place: mixed-use crawlers, which blend search, agent use and training under a single user-agent, account for more than 36% of crawler activity on their network. In other words, more than a third of the automated traffic reaching you cannot be cleanly sorted by purpose.

Who actually honours robots.txt

robots.txt is a request, not an enforcement mechanism. A server that receives a request from a user-agent which has declared that it ignores the file still serves the content. This is not a theoretical scenario: OpenAI’s own documentation states that robots.txt rules may not apply to ChatGPT-User, because the request follows an explicit action by a human user. Perplexity’s documentation says much the same about Perplexity-User, noting that it generally does not follow robots.txt for the same reason. Anthropic states that all three of its bots honour it.

The reasoning behind that behaviour is that the request counts as a human action rather than a crawl. You can argue with the reasoning, but in practice you should treat it as an operational fact: if you want a block that is genuinely enforced, robots.txt is not the tool. The tool sits at the CDN or firewall layer — Cloudflare, Fastly, or a WAF rule that keys on user-agent and IP range.

Crawl-to-refer ratios — why the trade is no longer symmetrical

The claim that crawling pays for itself in return traffic is historically true and numerically wrong. In Cloudflare’s analysis of 28 August 2025, the ratio of pages crawled per single referral back was roughly 50,000 to 1 for Anthropic, 887 to 1 for OpenAI and 118 to 1 for Perplexity. For traditional Google Search the ratio is in the single digits.

Those numbers do not support one conclusion. They say the old bargain — you supply content, the engine sends visitors — no longer describes what happens at the training layer. They do not say visibility in answer engines is worthless, because the referrals that do arrive convert better on average than ordinary search traffic: the user arrives after the system has already filtered on their behalf and recommended you by name.

Three viable positions, and who each one suits

Fully open. No blocks at all. Suits marketing sites, professional services and B2B that sell on trust and expertise. Your content is sales support material, not the asset itself; the more models that know you, the higher the odds one mentions you. The only cost is bandwidth.

Selective separation. Block training, allow answering. Suits most sites with meaningful original content: publishing, research, knowledge bases, product documentation. You pay for the layer that returns users to you and not for the layer that does not. This is the position I recommend as a default.

Full block. Block everything and enforce it at the CDN. Suits cases where the content is the product — archives, paid databases, image libraries — or where a contractual or regulatory obligation applies. Know going in that you are giving up presence in answer engines, not just training.

A sample robots.txt — selective separation

Three notes on this file. First, user-agent blocks do not accumulate: a crawler reads only the most specific block that matches it, so the restrictions under User-agent: * do not apply to any bot that has a block of its own. Second, Allow: / is technically redundant — the absence of a disallow is sufficient — but it documents intent, and that is worth something when somebody else reads the file a year from now. Third, there is no line for Googlebot, and that is deliberate.

How to verify it actually works

robots.txt testers show you what the file says, not what happens. The only verification that counts is your server access log. Three things worth measuring two weeks in:

1. Whether the blocked crawlers actually stopped. Filter the log by user-agent and count requests before and after. A partial drop means the rule is not being matched.
2. What response code the allowed crawlers receive. This is where the most common failure hides: a firewall, security plugin or bot-management layer returns 403 or a challenge to legitimate bots without anyone configuring it. The file allows; the server blocks.
3. What size the response is. A 200 carrying 4KB instead of 60KB means the bot received an HTML shell with no content — an entirely different problem, which I cover separately in the post on what an AI agent sees when it reaches your site.

One final point that saves wasted effort: on 10 July 2026 Google updated its official documentation to state explicitly that Google Search does not use llms.txt files and ignores them. If your bot-management plan started with creating an llms.txt, move it to the bottom of the list. It does no harm, but it also does not do what most guides promise.

Frequently asked questions