Many site owners treat robots.txt as a way to say “don’t use my content for AI”. Google has now explained, in plain words, why that is not what the file does. In the second episode of Search Central’s video series The Non-Canonical Cabinet, published on October 22, 2026, Gary Illyes and Cherry Prommawin walked through what robots.txt can and cannot control.
What Google said
- robots.txt is an access protocol. Under RFC 9309, the standard behind it, robots.txt tells a crawler which URLs it may request. It says nothing about what happens to the content once it has been fetched: indexing, summarizing or training a model are outside its scope.
- To keep an AI crawler out, name it. If you want to block one crawler and allow others, you must add a group with that crawler’s user-agent token, such as GPTBot or ClaudeBot.
- Invented directives are ignored. The episode showed a made-up rule,
No-AI-Chicken-Training, as an example of a line parsers skip. Google supports only four fields: User-agent, Allow, Disallow and Sitemap. - Other crawlers may behave differently. RFC 9309 lets crawlers decide what to do with lines outside the protocol, so the effect of any extra rule depends on each crawler.
- Encode non-English paths. Google’s parser handles raw characters, but Prommawin recommended percent-encoding non-English paths because older crawlers can get confused.
So how do you control AI use of your content?
Today there is no standard way in robots.txt to separate “you may crawl this” from “you may use it for training”. What exists is a set of crawler-specific controls:
- Block the training crawlers by name. OpenAI (GPTBot), Anthropic (ClaudeBot), Common Crawl (CCBot) and others run separate crawlers for training. Blocking them in robots.txt stops them from fetching your pages, if they respect the file.
- Use Google-Extended for Gemini. Google-Extended is not a crawler with its own user agent but a token Google reads to decide whether content crawled by Googlebot can be used to train future Gemini models and for grounding. It does not affect Search inclusion or rankings.
- Keep search and user bots if you want citations. Blocking AI search bots such as OAI-SearchBot or PerplexityBot means assistants can rarely cite you.
- Enforce at the edge if you need a lock. robots.txt is a request, not a barrier. CDNs and firewalls can block bots that ignore it. In our crawl of the top 1 million websites, 38.9% sit behind Cloudflare, which offers AI bot blocking.
Illyes has also co-written an IETF Internet-Draft with Mozilla’s Martin Thomson that would add a Content-Usage rule for stating AI usage preferences. It is still a draft and not a standard, so no crawler is required to honor it yet.
What the data says about current practice
In our study of robots.txt in the top websites, GPTBot is already named in more files than Googlebot, and some sites still try to block AI with unsupported lines. Check how your own file treats each AI crawler with our AI Bot Checker, test any rule with the robots.txt tester, and read our complete robots.txt guide.
Frequently asked questions
Frequently asked questions
Can robots.txt stop AI from training on my content?
Only indirectly. robots.txt can stop crawlers that respect it from fetching your pages, but it cannot set rules about how content already fetched is used.
Does Google-Extended remove my site from AI Overviews?
No. Google-Extended controls whether content can be used to train Gemini models and for grounding. It does not affect Google Search inclusion or rankings.
Which robots.txt directives does Google support?
User-agent, Allow, Disallow and Sitemap. Other lines, including invented AI rules, are ignored by Google.
What is the Content-Usage proposal?
An IETF draft co-written by Google’s Gary Illyes that would let sites state AI usage preferences in robots.txt. It is not a standard yet.