robots.txt is one of the oldest files on the web, and it has never been more important: it is now the main way websites talk to AI crawlers. We analyzed the robots.txt files of the 6,831 websites in the world’s top 10,000 domains (Tranco list 8PX8V) to see what they contain, how they have changed and which mistakes are most common. This study uses the same data as our study of AI bots and the top 10,000 websites.
Key findings
- 89.5% of top websites have a robots.txt file. The other 10.5% have none, which means every crawler may access everything.
- AI has taken over robots.txt. 23.4% of the files name at least one AI crawler, and 12 of the 13 most named user agents are AI bots.
- GPTBot is named more often than Googlebot: 17.6% of the files mention OpenAI’s crawler, compared with 9.2% for Googlebot.
- 71.1% declare an XML sitemap, and 25.3% declare more than one.
- Old habits persist: 11.9% use Crawl-delay, which Google ignores, 4.8% use the Yandex-only Host directive, and 0.4% still use noindex in robots.txt, unsupported by Google since 2019.
- Some sites shut out search engines: 2.2% block Googlebot completely and 2.9% block Bingbot.
The bots everyone is talking to
A few years ago, a typical robots.txt only had a general group for every bot and maybe a rule for Googlebot. Today the most named user agents are AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and Bytespider. Many of these lists are copied from templates and get outdated quickly: 1.2% of the files block Anthropic only through its old tokens (anthropic-ai or Claude-Web), without naming its current crawler, ClaudeBot.
The bots named most often in robots.txt
% of 6,114 robots.txt files that name each user agent
Most files also have a general group for every bot (User-agent: *).
What is inside the files
The median robots.txt is small, 649 bytes and 23 non-empty lines, but the range is huge: 10% of the files are larger than 6 KB and the longest has 14,086 lines. None of the files we analyzed reached Google’s 500 KiB limit, beyond which Google ignores the rest of the file.
What is inside robots.txt files
% of 6,114 robots.txt files
The most common mistakes
- Rules Google ignores. Crawl-delay (11.9%), Host (4.8%) and Clean-param (3.2%) only work in some other search engines. They are harmless, but they give a false sense of control over Googlebot.
- Unsupported noindex. 0.4% of the files try to keep pages out of Google with a noindex line. Google stopped supporting it in 2019: use a meta robots tag instead.
- Broken sitemap lines. 0.6% use a relative sitemap URL, which is invalid, and 1.5% point to an http:// sitemap.
- Unknown or malformed lines. 5.4% of the files contain lines crawlers cannot understand, from typos to directives that do not exist.
- Blocked resources. 1.7% block CSS or JavaScript for Googlebot, which can prevent Google from rendering pages correctly.
A clean robots.txt for 2026
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# AI search: allowed. AI training: your choice.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap_index.xmlTest your own file with our robots.txt tester, see how AI bots are treated with the AI Bot Checker, and read the complete robots.txt guide.
Methodology
We started from the top 10,000 domains of Tranco list 8PX8V and kept the 6,971 that serve a website. 6,831 had a readable robots.txt or none at all; 6,114 had a robots.txt file, which is the base for the percentages about file contents. Rules were evaluated following RFC 9309. “Blocks Googlebot completely” means Googlebot cannot reach any typical content page. “Blocks CSS or JS” means Googlebot can crawl a typical page but not a typical script or stylesheet. Data collected on October 17–18, 2026.
How to cite: SEOOn.page (2026). The State of robots.txt (2026). https://seoon.page/research/state-of-robots-txt-2026/