When people talk about blocking AI bots, they usually mean one thing: stopping AI companies from using their content. But the major AI companies now run several crawlers, each with a different job. Blocking the wrong one can make you invisible in AI answers without protecting your content at all.
Three jobs, three kinds of crawler
- Training crawlers collect content that may be used to train future models. Examples: GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, and the Google-Extended and Applebot-Extended tokens.
- Search crawlers build the index an assistant searches when it answers a question with live web results. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- User-triggered fetchers load a page when a person asks the assistant to read it. Examples: ChatGPT-User, Claude-User, Perplexity-User.
What happens when you block a training crawler
Your new content stops being collected for future training runs by that company. Your site can still be found and cited through its search crawler, and the model can still know about you from other sources, such as other websites that mention you. Content collected before the block is not removed from existing models.
What happens when you block a search crawler
Your pages drop out of that assistant’s live search results. When users ask ChatGPT, Claude or Perplexity a question your page could answer, the assistant will cite a competitor instead. For most businesses, this is the more expensive block, because AI answers increasingly send referral traffic and shape purchase decisions.
What happens when you block a user fetcher
When a user pastes your URL into an assistant and asks for a summary, the assistant may not be able to read the page. Note that some vendors say their user-triggered fetchers may not follow robots.txt the same way, because a human made the request.
The data: training is blocked more than search
In our study of 344 leading websites, websites blocked training crawlers more often than search crawlers: GPTBot was blocked by 15.3% of the sites and OAI-SearchBot by 10.2%; ClaudeBot by 15.6% and Claude-SearchBot by 8.9%. Yet only 5% of the sites deliberately blocked training while allowing search. Most simply copied long lists of bots into a single Disallow group.
The special case of Google
Google does not have a separate AI search crawler. AI Overviews and AI Mode use the same index as Google Search, crawled by Googlebot. Google-Extended is a token that only controls the use of your content for Gemini training and grounding. So you cannot block AI Overviews without blocking Google Search, but you can opt out of Gemini training without any ranking impact.
A sensible default
For most websites the best balance is to allow search crawlers and user fetchers, and decide case by case about training crawlers. You can generate this setup in one click with the Selective preset of our robots.txt tester and generator, and verify the result with the AI Bot Checker.
Frequently asked questions
Is GPTBot used for ChatGPT search?
No. ChatGPT search relies on OAI-SearchBot. GPTBot collects data for training.
If I block ClaudeBot, can Claude still cite my site?
Yes, as long as Claude-SearchBot and Claude-User are allowed.
Do all AI companies separate training and search?
The largest ones do, but not all. Smaller crawlers often use a single user agent for everything, so check each vendor’s documentation.