The Complete List of AI Crawlers and User Agents (2026)

The complete list of AI crawlers and user agents 2026

Written by

Mauricio Rodriguez

in

AI companies send crawlers to read the web for three different reasons: to collect training data, to build a search index for their assistants, and to fetch a page when a user asks about it. Each crawler identifies itself with a user agent token that you can allow or block in robots.txt. This list covers the crawlers that matter in 2026, grouped by company.

Want to know which of them your site blocks right now? Run your domain through our free AI Bot Checker.

How to read this list

  • Training: collects content to train or improve AI models. Blocking it keeps your content out of future training data, but does not remove you from AI answers.
  • Search: builds the index an AI assistant uses to find and cite sources. Blocking it makes you much less likely to be cited.
  • User request: fetches a specific page because a person asked the assistant to read or summarize it. Some vendors state that these fetchers may not follow robots.txt the same way, because a human triggered the request.

OpenAI (ChatGPT)

  • GPTBot (training): crawls public pages that may be used to train OpenAI’s models.
  • OAI-SearchBot (search): indexes pages so they can appear and be linked in ChatGPT search results.
  • ChatGPT-User (user request): visits a page when a ChatGPT user or a custom GPT asks for it.

OpenAI documents its crawlers and lets site owners control each one separately. Blocking GPTBot does not block OAI-SearchBot.

Anthropic (Claude)

  • ClaudeBot (training): collects public content that may be used to train Claude models.
  • Claude-SearchBot (search): improves the quality of Claude’s search results.
  • Claude-User (user request): fetches pages when a Claude user asks a question that needs them.
  • anthropic-ai: an older token that many robots.txt files still list. Keep it if you already block it, but use the three tokens above for new rules.

Google (Gemini and AI Overviews)

  • Google-Extended (training): not a separate crawler but a robots.txt token. It tells Google whether content crawled by Googlebot can be used to train and ground Gemini models.
  • Googlebot: Google’s main search crawler. AI Overviews and AI Mode in Google Search use the regular search index, so they are controlled by Googlebot, not by Google-Extended.

This is the most misunderstood point in AI crawler control: blocking Google-Extended does not remove your pages from AI Overviews, and blocking Googlebot removes you from Google Search entirely.

Perplexity

  • PerplexityBot (search): indexes pages so they can be surfaced and cited in Perplexity answers.
  • Perplexity-User (user request): fetches a page when a user asks a question. Perplexity states that this fetcher generally ignores robots.txt, because the request comes from a person.

Microsoft (Copilot)

Microsoft Copilot relies on the Bing index, crawled by Bingbot. There is no separate Copilot training token in robots.txt: if you block Bingbot you also leave Bing search.

Apple, Meta, Amazon and others

  • Applebot-Extended (Apple, training): a token that controls whether content crawled by Applebot can train Apple’s foundation models. Applebot itself powers Siri and Spotlight suggestions.
  • meta-externalagent (Meta, training): crawls content for Meta’s AI models. Meta also uses meta-externalfetcher for user-triggered fetches.
  • Amazonbot (Amazon): used for Amazon services including Alexa answers.
  • CCBot (Common Crawl): builds the free Common Crawl dataset, one of the most widely used sources for training language models.
  • Bytespider (ByteDance, training): crawls for ByteDance’s models. It is one of the most blocked crawlers.
  • cohere-ai (Cohere), MistralAI-User (Mistral, user request) and DuckAssistBot (DuckDuckGo, search) complete the list of crawlers we track.

How many websites block them?

In our study of 344 leading websites, GPTBot, ClaudeBot, CCBot and Bytespider were each blocked by about 15% of the sites, while AI search bots were blocked less often. News media were by far the most restrictive sector.

How to block or allow AI crawlers

List the tokens you want to control in their own group of rules. A common setup allows AI search and blocks AI training:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Build and test your own version with our robots.txt tester and generator. Remember that robots.txt is a request: a few crawlers ignore it, and a CDN or firewall rule can block bots regardless of what your robots.txt says.

Frequently asked questions

Which AI crawler should I never block?

If you want to be cited in AI answers, keep the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed, and never block Googlebot or Bingbot unless you want to leave classic search too.

Are user agent strings enough to identify a real AI crawler?

No. Anyone can fake a user agent. The major vendors publish IP ranges so you can verify that a request really comes from them.

How often does this list change?

Several times a year. We update this page whenever a major vendor launches or renames a crawler.

More posts