Every website now faces a question that did not exist a few years ago: should AI companies be allowed to read your content? There is no single right answer. It depends on how your site makes money and how much you depend on being found. This guide helps you choose a strategy and gives you the exact robots.txt rules for each one.
What you gain and what you lose
- If you allow AI bots, your pages can be cited and linked in AI answers, and assistants can describe your brand accurately. The cost is that your content may be used to train models without direct compensation.
- If you block AI bots, you keep more control over your content, but you become invisible or poorly represented in a growing share of search journeys.
The key is that AI companies use separate crawlers for training and for search, so you do not have to choose all or nothing.
The three strategies
1. Open: allow everything
Best for businesses that want maximum visibility: SaaS, ecommerce, local businesses, agencies, hosting and telecom companies. You want assistants to know your products and recommend you.
User-agent: *
Allow: /2. Selective: allow AI search, block AI training
Best for most content websites: blogs, niche sites, B2B publishers. You stay eligible for citations in ChatGPT, Claude and Perplexity, while opting out of model training.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /3. Closed: block all AI bots
Best for publishers whose content is the product and who negotiate licensing deals: news media, paid databases, course platforms. Classic search engines stay allowed.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /What other websites do
In our study of 344 leading websites, 1 in 5 websites blocked at least one AI bot, but only 5% used the selective strategy. News media were the most restrictive, with more than half blocking AI crawlers. Telecom, energy and hosting companies blocked almost none and were the most likely to publish an llms.txt file.
Mistakes to avoid
- Blocking Google-Extended to leave AI Overviews. It does not work: AI Overviews use the normal Google Search index. Google-Extended only affects Gemini training and grounding.
- Forgetting that a named group replaces the default one. If you add a group for GPTBot, it stops following your User-agent: * rules. Repeat any private paths you need.
- Trusting robots.txt alone. It is a public request. Some crawlers ignore it, and user-triggered fetchers may not follow it.
- Ignoring your CDN. Cloudflare and other providers can block AI bots at the network level, which can contradict what your robots.txt says.
How to check your current setup
Enter your domain in our AI Bot Checker to see, bot by bot, what your robots.txt allows. Then use the robots.txt tester and generator to create a new file with one of the three presets above.
Frequently asked questions
Will blocking AI bots hurt my Google rankings?
No. Google rankings depend on Googlebot. Blocking GPTBot, ClaudeBot or Google-Extended does not change how Google Search ranks your pages.
Can I block AI bots from only part of my site?
Yes. Use Disallow rules with specific paths, such as Disallow: /premium/, inside the group of each AI bot.
Does blocking AI bots remove content they already collected?
No. It only affects future crawling. Content already used for training stays in existing models.