What this AI bot checker does
Enter any domain and the tool reads its public robots.txt the same way an AI crawler does. It then tells you, bot by bot, whether OpenAI, Anthropic, Perplexity, Google, Apple, Meta and other AI companies are allowed to crawl the site. It also checks whether the site publishes an llms.txt file.
We evaluate every rule following RFC 9309, the robots.txt standard used by Google: each bot follows its own group of rules, or the default User-agent: * group if it has none, and the most specific rule wins.
Training bots, search bots and user bots
AI companies run different crawlers for different jobs, and blocking each one has a different effect:
- Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended) collect content to train AI models. Blocking them keeps your content out of future training data.
- Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so AI assistants can find and cite them in answers. Blocking them means you will rarely be cited.
- User bots (ChatGPT-User, Claude-User, Perplexity-User) fetch a page when a person asks the assistant about it. Blocking them stops the assistant from reading your page on demand.
How to read the results
- Allowed: the bot can crawl your content.
- Partially blocked: the bot has its own rules that block some sections, but not the whole site.
- Blocked: the bot cannot reach any content page. A site that only allows its home page counts as blocked.
- Rule: whether the bot is named in its own group or simply follows the default rules for every bot.
Which strategy should you choose?
For most websites that live on traffic and visibility, we recommend a selective strategy: allow AI search and user bots so you can be cited in AI answers, and decide separately about training bots. Publishers whose content is the product often block training bots. You can create a selective robots.txt with our robots.txt generator.
Want to know what other websites do? See our research on AI bots, where we analyzed hundreds of sites.
Limitations
robots.txt is a public request, not a lock. Most large AI companies say they respect it, but some bots do not. A CDN or firewall (for example Cloudflare’s AI bot blocking) can also block bots at the server level, which this tool does not see.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
Not completely. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a user asks. To stop ChatGPT from citing you, you would need to block those as well.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your content is used to train and ground Gemini models. Googlebot, which crawls for Google Search, is a separate user agent.
What happens if a site has no robots.txt?
Every crawler, including all AI bots, is allowed to crawl the entire site.
How often is the data refreshed?
Results are cached for 30 minutes. Run the check again later to see recent changes.