Category: AI Search

When people talk about blocking AI bots, they usually mean one thing: stopping AI companies from using their content. But the major AI companies now run several crawlers, each with a different job. Blocking the wrong one can make you invisible in AI answers without protecting your content at all.

Three jobs, three kinds of crawler

• Training crawlers collect content that may be used to train future models. Examples: GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, and the Google-Extended and Applebot-Extended tokens.
• Search crawlers build the index an assistant searches when it answers a question with live web results. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
• User-triggered fetchers load a page when a person asks the assistant to read it. Examples: ChatGPT-User, Claude-User, Perplexity-User.

What happens when you block a training crawler

Your new content stops being collected for future training runs by that company. Your site can still be found and cited through its search crawler, and the model can still know about you from other sources, such as other websites that mention you. Content collected before the block is not removed from existing models.

What happens when you block a search crawler

Your pages drop out of that assistant’s live search results. When users ask ChatGPT, Claude or Perplexity a question your page could answer, the assistant will cite a competitor instead. For most businesses, this is the more expensive block, because AI answers increasingly send referral traffic and shape purchase decisions.

What happens when you block a user fetcher

When a user pastes your URL into an assistant and asks for a summary, the assistant may not be able to read the page. Note that some vendors say their user-triggered fetchers may not follow robots.txt the same way, because a human made the request.

The data: training is blocked more than search

In our study of 344 leading websites, websites blocked training crawlers more often than search crawlers: GPTBot was blocked by 15.3% of the sites and OAI-SearchBot by 10.2%; ClaudeBot by 15.6% and Claude-SearchBot by 8.9%. Yet only 5% of the sites deliberately blocked training while allowing search. Most simply copied long lists of bots into a single Disallow group.

The special case of Google

Google does not have a separate AI search crawler. AI Overviews and AI Mode use the same index as Google Search, crawled by Googlebot. Google-Extended is a token that only controls the use of your content for Gemini training and grounding. So you cannot block AI Overviews without blocking Google Search, but you can opt out of Gemini training without any ranking impact.

A sensible default

For most websites the best balance is to allow search crawlers and user fetchers, and decide case by case about training crawlers. You can generate this setup in one click with the Selective preset of our robots.txt tester and generator, and verify the result with the AI Bot Checker.

Frequently asked questions

Is GPTBot used for ChatGPT search?

No. ChatGPT search relies on OAI-SearchBot. GPTBot collects data for training.

If I block ClaudeBot, can Claude still cite my site?

Yes, as long as Claude-SearchBot and Claude-User are allowed.

Do all AI companies separate training and search?

The largest ones do, but not all. Smaller crawlers often use a single user agent for everything, so check each vendor’s documentation.

llms.txt is a plain text file written in Markdown and published at the root of a website, at /llms.txt. It gives large language models and AI agents a short summary of the site and a curated list of links to its most useful pages. The idea was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI, on llmstxt.org.

Why llms.txt exists

Modern websites are hard for AI systems to read: navigation menus, scripts, cookie banners and ads take up space, and a model has a limited context window. llms.txt solves this by offering a clean, human-written entry point: what the site is, and where to find the information that matters.

It is often compared to robots.txt and sitemaps, but it does a different job. robots.txt says what bots may crawl. A sitemap lists every URL. llms.txt explains and curates.

The llms.txt format

• H1 title with the name of the site or project. This is the only required part.
• Blockquote with a short summary of what the site is about.
• Optional paragraphs with key details an assistant should know.
• H2 sections, each with a Markdown list of links in the form - [Title](URL): description.
• An Optional section for secondary links that can be skipped when space is short.

A complete example

# SEOOn.page

> Free SEO tools and original research for the AI search era.

SEOOn.page offers free tools to check how search engines and AI assistants see a website.

## Tools

- [AI Bot Checker](https://seoon.page/tools/ai-bot-checker/): Check which AI crawlers a site allows
- [llms.txt Generator](https://seoon.page/tools/llms-txt-generator/): Create and validate llms.txt files

## Research

- [AI Bots and Spain's Top Websites](https://seoon.page/research/ai-bots-spain-top-websites-2026/): Which sites block AI crawlers

## Optional

- [Blog](https://seoon.page/blog/)

Who uses llms.txt?

Adoption started with developer documentation and AI companies, and is spreading to other sectors. In our study of 344 leading websites, 15% of the websites already published an llms.txt file, with tech, hosting and telecom companies leading the way. News media, on the other hand, rarely publish one.

Does llms.txt help SEO or AI visibility?

Be realistic about it. Google has said it does not use llms.txt for Search, and no major AI company has confirmed that its crawlers read it systematically. Where it clearly helps is with AI agents and tools that read a site on demand, such as coding assistants and research agents, which can use it to find the right pages quickly.

Because it takes a few minutes to create and has no downside, many sites treat it as a low-cost bet on how AI agents will navigate the web.

llms.txt vs llms-full.txt

Some sites also publish llms-full.txt, a single file with the full text of their key pages in Markdown. It is useful for documentation sites, where an assistant may want everything in one request. For most websites, a short llms.txt is enough.

How to create your llms.txt

1. Write a one-sentence summary of your website.
2. Pick 5 to 20 pages that best explain your business: products, pricing, documentation, about, contact.
3. Use our llms.txt generator to turn them into a valid file.
4. Upload it to the root of your domain and check that it loads as plain text.
5. Validate it with the same tool and update it when your key pages change.

Frequently asked questions

Is llms.txt an official standard?

No. It is a community proposal, not a W3C or IETF standard, but its format is simple and widely copied.

Should llms.txt include every page?

No. That is what sitemaps are for. Keep llms.txt short and curated.

Can llms.txt block AI bots?

No. To control access, use robots.txt. llms.txt only describes your site.

Every website now faces a question that did not exist a few years ago: should AI companies be allowed to read your content? There is no single right answer. It depends on how your site makes money and how much you depend on being found. This guide helps you choose a strategy and gives you the exact robots.txt rules for each one.

What you gain and what you lose

• If you allow AI bots, your pages can be cited and linked in AI answers, and assistants can describe your brand accurately. The cost is that your content may be used to train models without direct compensation.
• If you block AI bots, you keep more control over your content, but you become invisible or poorly represented in a growing share of search journeys.

The key is that AI companies use separate crawlers for training and for search, so you do not have to choose all or nothing.

The three strategies

1. Open: allow everything

Best for businesses that want maximum visibility: SaaS, ecommerce, local businesses, agencies, hosting and telecom companies. You want assistants to know your products and recommend you.

User-agent: *
Allow: /

2. Selective: allow AI search, block AI training

Best for most content websites: blogs, niche sites, B2B publishers. You stay eligible for citations in ChatGPT, Claude and Perplexity, while opting out of model training.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

3. Closed: block all AI bots

Best for publishers whose content is the product and who negotiate licensing deals: news media, paid databases, course platforms. Classic search engines stay allowed.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

What other websites do

In our study of 344 leading websites, 1 in 5 websites blocked at least one AI bot, but only 5% used the selective strategy. News media were the most restrictive, with more than half blocking AI crawlers. Telecom, energy and hosting companies blocked almost none and were the most likely to publish an llms.txt file.

Mistakes to avoid

• Blocking Google-Extended to leave AI Overviews. It does not work: AI Overviews use the normal Google Search index. Google-Extended only affects Gemini training and grounding.
• Forgetting that a named group replaces the default one. If you add a group for GPTBot, it stops following your User-agent: * rules. Repeat any private paths you need.
• Trusting robots.txt alone. It is a public request. Some crawlers ignore it, and user-triggered fetchers may not follow it.
• Ignoring your CDN. Cloudflare and other providers can block AI bots at the network level, which can contradict what your robots.txt says.

How to check your current setup

Enter your domain in our AI Bot Checker to see, bot by bot, what your robots.txt allows. Then use the robots.txt tester and generator to create a new file with one of the three presets above.

Frequently asked questions

Will blocking AI bots hurt my Google rankings?

No. Google rankings depend on Googlebot. Blocking GPTBot, ClaudeBot or Google-Extended does not change how Google Search ranks your pages.

Can I block AI bots from only part of my site?

Yes. Use Disallow rules with specific paths, such as Disallow: /premium/, inside the group of each AI bot.

Does blocking AI bots remove content they already collected?

No. It only affects future crawling. Content already used for training stays in existing models.

AI companies send crawlers to read the web for three different reasons: to collect training data, to build a search index for their assistants, and to fetch a page when a user asks about it. Each crawler identifies itself with a user agent token that you can allow or block in robots.txt. This list covers the crawlers that matter in 2026, grouped by company.

Want to know which of them your site blocks right now? Run your domain through our free AI Bot Checker.

How to read this list

• Training: collects content to train or improve AI models. Blocking it keeps your content out of future training data, but does not remove you from AI answers.
• Search: builds the index an AI assistant uses to find and cite sources. Blocking it makes you much less likely to be cited.
• User request: fetches a specific page because a person asked the assistant to read or summarize it. Some vendors state that these fetchers may not follow robots.txt the same way, because a human triggered the request.

OpenAI (ChatGPT)

• GPTBot (training): crawls public pages that may be used to train OpenAI’s models.
• OAI-SearchBot (search): indexes pages so they can appear and be linked in ChatGPT search results.
• ChatGPT-User (user request): visits a page when a ChatGPT user or a custom GPT asks for it.

OpenAI documents its crawlers and lets site owners control each one separately. Blocking GPTBot does not block OAI-SearchBot.

Anthropic (Claude)

• ClaudeBot (training): collects public content that may be used to train Claude models.
• Claude-SearchBot (search): improves the quality of Claude’s search results.
• Claude-User (user request): fetches pages when a Claude user asks a question that needs them.
• anthropic-ai: an older token that many robots.txt files still list. Keep it if you already block it, but use the three tokens above for new rules.

Google (Gemini and AI Overviews)

• Google-Extended (training): not a separate crawler but a robots.txt token. It tells Google whether content crawled by Googlebot can be used to train and ground Gemini models.
• Googlebot: Google’s main search crawler. AI Overviews and AI Mode in Google Search use the regular search index, so they are controlled by Googlebot, not by Google-Extended.

This is the most misunderstood point in AI crawler control: blocking Google-Extended does not remove your pages from AI Overviews, and blocking Googlebot removes you from Google Search entirely.

Perplexity

• PerplexityBot (search): indexes pages so they can be surfaced and cited in Perplexity answers.
• Perplexity-User (user request): fetches a page when a user asks a question. Perplexity states that this fetcher generally ignores robots.txt, because the request comes from a person.

Microsoft (Copilot)

Microsoft Copilot relies on the Bing index, crawled by Bingbot. There is no separate Copilot training token in robots.txt: if you block Bingbot you also leave Bing search.

Apple, Meta, Amazon and others

• Applebot-Extended (Apple, training): a token that controls whether content crawled by Applebot can train Apple’s foundation models. Applebot itself powers Siri and Spotlight suggestions.
• meta-externalagent (Meta, training): crawls content for Meta’s AI models. Meta also uses meta-externalfetcher for user-triggered fetches.
• Amazonbot (Amazon): used for Amazon services including Alexa answers.
• CCBot (Common Crawl): builds the free Common Crawl dataset, one of the most widely used sources for training language models.
• Bytespider (ByteDance, training): crawls for ByteDance’s models. It is one of the most blocked crawlers.
• cohere-ai (Cohere), MistralAI-User (Mistral, user request) and DuckAssistBot (DuckDuckGo, search) complete the list of crawlers we track.

How many websites block them?

In our study of 344 leading websites, GPTBot, ClaudeBot, CCBot and Bytespider were each blocked by about 15% of the sites, while AI search bots were blocked less often. News media were by far the most restrictive sector.

How to block or allow AI crawlers

List the tokens you want to control in their own group of rules. A common setup allows AI search and blocks AI training:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Build and test your own version with our robots.txt tester and generator. Remember that robots.txt is a request: a few crawlers ignore it, and a CDN or firewall rule can block bots regardless of what your robots.txt says.

Frequently asked questions

Which AI crawler should I never block?

If you want to be cited in AI answers, keep the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed, and never block Googlebot or Bingbot unless you want to leave classic search too.

Are user agent strings enough to identify a real AI crawler?

No. Anyone can fake a user agent. The major vendors publish IP ranges so you can verify that a request really comes from them.

How often does this list change?

Several times a year. We update this page whenever a major vendor launches or renames a crawler.