AI crawlers identify themselves with a user agent token, such as GPTBot or ClaudeBot, that site owners can allow or block in robots.txt. The major AI companies run separate crawlers for model training, AI search and user-triggered requests, so each one can be controlled on its own.
See our complete list of AI crawlers and check which ones your site blocks with the AI Bot Checker.