Open a server log from the last year and you will find traffic that did not exist before. GPTBot working through a few thousand URLs overnight. ClaudeBot grinding steadily through your archive. PerplexityBot arriving seconds after someone asked a question about your category. Most teams notice the bandwidth bill before they understand the traffic.
The reflex is to block everything. That reflex is usually wrong, and it is wrong for a specific reason: the bots in your logs are not doing the same job. Some are collecting training data. Some are building the index that decides whether an AI assistant cites you. Some are fetching a page right now because a human asked for it. Blocking them has three completely different consequences.
This is the reference: who each bot is, what its user agent looks like, how to confirm it is real, and what you give up by blocking it.
Three kinds of AI crawler, and why the distinction matters
Almost every mistake in this area comes from treating "AI bot" as one category. There are three, and they map to three different business outcomes.
Training crawlers collect content to train or fine-tune models. GPTBot, ClaudeBot, CCBot and Bytespider are the main ones. Blocking them keeps your content out of future training runs. It does not remove you from models already trained, and it has no effect on whether an assistant cites you today. This is a rights and licensing decision, not a traffic decision.
Search-index crawlers build the retrieval index an assistant queries when a user asks a question. OAI-SearchBot, Claude-SearchBot and PerplexityBot sit here. Blocking these is the equivalent of blocking Googlebot. You disappear from the answer, along with the link. This is almost always the wrong call.
User-triggered fetchers load a single URL in real time because a person asked the assistant to read it. ChatGPT-User, Claude-User and Perplexity-User do this. They are not crawling - they are a browser with an unusual user agent. Blocking them means that when a prospect pastes your pricing page into ChatGPT and asks what it costs, the assistant tells them it could not access the page.
Roughly half of all AI crawling on the web is for training, and only a small fraction is search-related. That ratio is exactly why blanket blocking is a bad trade: you cut a large volume of training crawl, and in doing so you also cut the small, high-value slice that produces citations and referral clicks.
The user agents, by operator
User-agent strings change. Match on the product token - the short identifier - not the full string, and match case-insensitively.
OpenAI
GPTBot is the training crawler. Full string looks like Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot.
OAI-SearchBot builds the index behind ChatGPT search. This is the one that determines whether you appear in ChatGPT's cited sources. OpenAI has stated it is not used for training.
ChatGPT-User is the live fetcher. It fires when a user asks ChatGPT to visit a link, or when a plugin or action calls out to your site. Low volume, high intent.
OpenAI publishes IP ranges for each as separate JSON files on its site - the correct way to verify them.
Anthropic
ClaudeBot is the general crawler, identified as ClaudeBot/1.0; +claudebot@anthropic.com. It became one of the highest-volume AI crawlers on the web in 2026, at roughly 20% of all AI crawl traffic. If your bandwidth graph has a new shape, ClaudeBot is a likely cause.
Claude-User handles user-initiated fetches from Claude, and Claude-SearchBot handles search indexing. You will still see legacy tokens like anthropic-ai and Claude-Web in old robots.txt files; harmless to keep, but no longer the primary identifiers.
Perplexity
PerplexityBot indexes pages so they can be surfaced and cited in Perplexity answers. Perplexity-User fetches a page in response to a specific user request. Perplexity has stated publicly that user-triggered fetches are not automated crawling and are therefore not governed by robots.txt in the same way - worth knowing before you assume a disallow rule covers it.
Cloudflare has also publicly accused Perplexity of crawling from undeclared user agents and rotating IPs after being blocked - a good argument for verifying identity at the network layer rather than trusting a header.
This is where most people get it wrong. Google-Extended is not a crawler. It never appears in your logs. It is a robots.txt token only, and it controls whether content Googlebot has already fetched can be used to train and ground Gemini. Disallowing it has no effect on Google Search rankings, and no effect on AI Overviews, which are served from the normal Search index by Googlebot.
GoogleOther is a real crawler used for one-off fetches and internal research and development, separate from the Search index. Googlebot remains the fetcher behind AI Overviews and AI Mode, which is why blocking Googlebot to avoid AI features is self-defeating.
The rest of the field
Bytespider belongs to ByteDance. It is high volume and has a poor reputation for respecting robots.txt. Verify it at the network layer if you care.
Amazonbot crawls for Alexa and Amazon's shopping assistants. Applebot powers Siri and Spotlight, while Applebot-Extended is the robots.txt opt-out token for Apple Intelligence training - same pattern as Google-Extended, and likewise not a real crawler.
CCBot is Common Crawl, a non-profit archive. It does not train anything itself, but its corpus feeds a large number of models. Blocking CCBot is an indirect training opt-out that only applies going forward.
meta-externalagent is Meta's crawler for training and product features, with meta-externalfetcher covering user-triggered loads. Bingbot is worth remembering because Microsoft Copilot is built on the Bing index - blocking Bingbot removes you from Copilot.
Below that sit data brokers and scrapers: Diffbot, Omgili, ImagesiftBot, Timpibot and others. Mostly a bandwidth question.
Finding them in your logs
You do not need tooling to start. If you have raw access logs in combined format, awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -50 gives you a ranked list of user agents. Anything with bot, crawler, GPT, Claude, AI or spider in the token is worth a second look.
What you want is four numbers per bot: request count, unique URLs, bytes served, and the status-code split. The status codes matter most. A high 404 rate means the crawler is working from a stale link graph. A high 403 or 429 rate means your CDN is already blocking bots you may not have intended to block - common on Cloudflare and Akamai, where bot rules were set years ago and never revisited.
Then check whether crawl volume matches content value. If ClaudeBot is spending most of its requests on paginated tag archives and faceted URLs, that is a crawl-efficiency problem, and you fix it the way you would for Googlebot.
If you would rather not write awk, the AI Crawler Analyzer takes a raw server or CDN log export and returns a per-bot breakdown with reverse-DNS spoof detection already applied, which saves the manual verification pass described next.
Verifying a bot is genuine
The user-agent header is a string an attacker types. It proves nothing. Cloudflare found that roughly 31% of traffic claiming to be GPTBot was actually other scrapers wearing the name - nearly a third of what looks like OpenAI in your logs is not OpenAI.
There are two legitimate verification methods, and which one applies depends on the operator.
Forward-confirmed reverse DNS. This is the standard technique, used by Google and Bing. Take the request IP and run a reverse lookup to get a hostname. Confirm the hostname ends in an operator-controlled domain - googlebot.com, google.com or googleusercontent.com for Google, search.msn.com for Bing. Then run a forward lookup on that hostname and confirm it resolves back to the original IP. Both directions must agree. A reverse lookup alone is not enough, because PTR records can be set by whoever controls the IP block.
In shell terms, that is dig -x 66.249.66.1 +short followed by dig +short <returned-hostname> and comparing the result to the IP you started with.
Published IP ranges. OpenAI, Anthropic and Perplexity publish the address ranges their bots crawl from as JSON files on their own domains. For these operators, matching the request IP against the current published ranges is the authoritative check. Fetch the range files on a schedule rather than hardcoding them - they change.
Cloudflare enabled AI crawler verification by default on free plans in 2026, so if you are behind Cloudflare much of this is already happening. Check what your bot rules actually do before building your own.
The practical upshot: never block or rate limit on the user-agent string alone. Verify first. Unverified traffic claiming to be a major AI bot is a scraper you never intended to permit, and it deserves harsher treatment than the real bot would.
Controlling access in robots.txt
Robots.txt is per-agent, and the matching rules trip people up constantly.
A compliant crawler obeys exactly one group - the most specific group whose user-agent token matches. Groups do not merge. So if you have a User-agent: * group with a list of disallows, and you add a User-agent: GPTBot group containing only Disallow: /private/, GPTBot now ignores every rule in the wildcard group. Your carefully built exclusions no longer apply to it. If you add a named group, restate everything that agent should still respect.
Token matching is a case-insensitive substring match against the product token, so User-agent: Claude would catch ClaudeBot, Claude-User and Claude-SearchBot at once. That is occasionally what you want and frequently not.
Robots.txt is also advisory. It is a request under RFC 9309, not access control. Bytespider and spoofers ignore it. Enforcement needs firewall or WAF rules keyed to verified identity - robots.txt for well-behaved bots, network rules for the rest.
One more thing: a disallow prevents crawling, not indexing of the URL itself, and it does not remove content from a model that already trained on it. Opting out is always forward-looking.
What blocking actually costs
The debate is usually framed as whether to let AI companies train on your content for free. That is a real question, but it is not the one your robots.txt answers unless you write it precisely.
Block GPTBot and you have opted out of training. You are still eligible for ChatGPT citations, because that runs through OAI-SearchBot. This is the surgical position, and for most publishers it is the right one.
Block everything OpenAI operates, and you have removed yourself from a surface where people are actively researching purchases. You do not get a penalty; you simply stop being one of the options the assistant can name. Your competitor gets named instead. That cost is invisible in your analytics - there is no impression count for an answer you were left out of.
Block user-triggered agents and you break something more immediate. A user explicitly asked to read your page. The assistant now reports that your site refused. That reads as a broken site, not a principled stance.
There is a legitimate counter-argument. If you sell access to content - a news archive, a research database, a recipe corpus monetised by display ads - AI answers substitute for your traffic rather than routing to it, and licensing or blocking is rational. Several large publishers have made that choice deliberately. Blocking is not always wrong. It should just be a decision with a number attached, not a default.
If you want to see where you currently stand before changing anything, the AI Visibility Grader shows how assistants currently describe and cite your site, which is the baseline you would be trading away.
What to actually do
Start by measuring. Pull thirty days of logs, split by bot, and verify identity before drawing conclusions. You will usually find two surprises: a bot you thought you were blocking is not, and a chunk of GPTBot traffic is fake.
Then set policy per category rather than per bot. Allow search-index crawlers everywhere you allow Googlebot. Allow user-triggered fetchers unconditionally - they represent a person who asked for you. Make training crawlers an explicit business decision, and write it down so the next person does not undo it.
Enforce at the network layer using verified identity, and reserve robots.txt for signalling intent to compliant bots. Rate limit rather than block when the issue is bandwidth rather than principle; most AI crawlers back off cleanly on a 429.
Finally, fix the boring things. AI crawlers are worse at JavaScript rendering than Googlebot, they follow stale link graphs, and they lean on structured data heavily when deciding what a page is about. Server-rendered content, clean canonicals and accurate JSON-LD markup do more for AI visibility than any robots.txt tuning. A technical audit will surface the rendering and crawl-waste issues that are quietly costing you far more than bot policy ever will.
Frequently asked questions
Should I block AI crawlers?
Not as a blanket policy. Block training crawlers if you have a commercial reason to protect your content, but keep search-index crawlers and user-triggered fetchers open - those produce citations and referral traffic. Blocking everything removes you from AI answers without removing your content from models already trained on it.
How do I tell if GPTBot in my logs is really OpenAI?
Match the request IP against OpenAI's published IP range files rather than trusting the user-agent header. Cloudflare found around 31% of GPTBot-labelled traffic was spoofed by other scrapers. For Google and Bing, use forward-confirmed reverse DNS instead: reverse-lookup the IP, check the hostname belongs to the operator, then forward-resolve it back to the same IP.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended is a robots.txt token that controls Gemini training and grounding only. It is not a crawler and never appears in logs. Your Search rankings and your appearance in AI Overviews are both driven by Googlebot, which Google-Extended does not touch.
Why is ClaudeBot using so much of my bandwidth?
ClaudeBot has become one of the highest-volume AI crawlers on the web, at roughly 20% of AI crawl traffic. If the cost is genuinely a problem, rate limit with a 429 rather than blocking outright, and check whether it is wasting requests on faceted or paginated URLs you could prune for every crawler at once.