How do the most popular websites in the world treat AI crawlers? We took the top 10,000 domains of the Tranco ranking, kept the 6,971 that are actual websites, and read the robots.txt file of each one to see how it treats 18 AI bots from OpenAI, Anthropic, Google, Perplexity, Apple, Meta and others. This is the global edition of our study of Spain’s top websites.
Key findings
- 17.4% of the world’s top websites fully block at least one AI bot in their robots.txt. In 15.5% of them the block is aimed at AI specifically, because Googlebot is still allowed.
- The most popular sites are the most defensive: 29.9% of the top 100 block an AI bot, compared with 16.8% of sites ranked 1,001 to 10,000.
- GPTBot is blocked by 12.5% of the websites, ClaudeBot by 11.4% and Common Crawl’s CCBot by 12.9%.
- 9.1% block at least one AI search bot, which makes them unlikely to be cited by ChatGPT, Claude or Perplexity.
- 15.4% publish an llms.txt file, a format first proposed in September 2024.
- The selective strategy is rare: only 6.3% block OpenAI’s training crawler while allowing its search crawler.
Which AI bots are blocked the most?
Training crawlers are blocked far more often than search crawlers, in the world and in Spain alike. The global top 10,000 is somewhat less restrictive than the leading Spanish websites, where 21.7% block at least one AI bot.
Websites that fully block each AI bot
% of websites, based on their robots.txt
Popularity and AI blocking
Large websites have more content to protect, more legal resources and more leverage to negotiate licensing deals with AI companies. The data reflects that: the share of sites that block AI bots grows as we move up the ranking.
The bigger the site, the more it blocks AI
% of websites in each Tranco ranking tier
By domain extension
Groups with at least 50 websites
| Extension | Sites | Blocks any AI bot | GPTBot | ClaudeBot | Any AI search bot | llms.txt |
|---|---|---|---|---|---|---|
| .com | 3,712 | 17.6% | 11.7% | 11.2% | 8.7% | 20.4% |
| country codes (ccTLD) | 1,922 | 19.6% | 15.6% | 13.5% | 10.6% | 10.2% |
| .org | 319 | 13.2% | 10.3% | 8.8% | 6.3% | 8.8% |
| other gTLDs | 317 | 15.1% | 12.9% | 12.3% | 10.4% | 9.5% |
| .net | 242 | 19.8% | 14.9% | 13.6% | 12% | 8.7% |
| .io | 110 | 6.4% | 4.5% | 3.6% | 4.5% | 32.7% |
| .edu | 107 | 2.8% | 1.9% | 0.9% | 1.9% | 3.7% |
| .gov | 102 | 5.9% | 2% | 2.9% | 2.9% | 1% |
What it means for your website
Blocking AI training crawlers is a legitimate choice, especially for publishers. But blocking AI search crawlers has a direct cost: those sites are left out of the answers that a growing share of users read instead of a list of links. For most businesses, the best balance is to allow AI search and decide separately about training.
# AI search: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# AI training: your choice
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Check your own site with the free AI Bot Checker, or read our guide to blocking AI bots.
Methodology
Sample. The top 10,000 domains of the Tranco list 8PX8V, a research-oriented ranking that combines several popularity sources and averages them over 30 days. Data collected in October 2026.
Websites only. Tranco also ranks infrastructure domains such as CDNs, APIs and ad servers. We kept the 6,971 domains whose home page responds, and excluded 3,029 that do not serve a website. Of those websites, 140 have a robots.txt we could not read because of bot protection, which leaves a valid sample of 6,831 websites. 717 of them have no robots.txt and count as allowing every bot.
Interpretation. Each robots.txt is evaluated following RFC 9309, the same logic Google uses. A bot counts as fully blocked when it cannot reach any typical content page. Sites that block Googlebot too are counted, and reported separately as blocking every crawler.
Limitations. robots.txt is a public request, not a guarantee, and some sites block bots at the CDN or firewall level instead. The ranking reflects popularity, not quality, and includes adult and regional sites. Results describe the situation in October 2026.
Open data
The full dataset, domain by domain, is available under a CC BY 4.0 license. Download the CSV.
How to cite: SEOOn.page (2026). AI Bots vs the World’s Top 10,000 Websites (2026). https://seoon.page/research/ai-bots-top-10000-websites-2026/
Journalists who need a specific cut of the data can contact us.