Which websites let ChatGPT, Claude, Perplexity and Gemini in, and which ones shut the door? We read the robots.txt file of 344 leading Spanish websites across 12 sectors, from newspapers and banks to retailers, universities and public administrations, and checked how each one treats 18 AI crawlers. Data collected on October 14, 2026.
Key findings
- 1 in 5 websites (21.7%) fully blocks at least one AI bot in its robots.txt.
- News media are the most defensive sector: 57.6% block at least one AI bot and 47.5% block OpenAI’s GPTBot.
- 15.3% already publish an llms.txt file, led by tech and hosting companies (43.8%) and telecom and energy (40.9%).
- 12.1% block at least one AI search bot (OAI-SearchBot, Claude-SearchBot or PerplexityBot), which makes them unlikely to be cited in AI answers.
- Only 5.1% use a selective strategy: they block OpenAI’s training crawler but allow its search crawler.
- Google-Extended is blocked half as often as GPTBot (7% vs 15.3%).
Which AI bots are blocked the most?
GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Bytespider (ByteDance) are the most blocked crawlers, each refused by about 15% of the websites. What they have in common: they collect content to train AI models. Googlebot is shown as a baseline: almost nobody blocks it.
Websites that fully block each bot
% of 314 websites, based on their robots.txt
“Fully blocked” means the robots.txt prevents the bot from reaching any content page, through its own rules or the default User-agent: * group.
Two opposite strategies: close the door or invite AI in
News media are the most defensive sector: 47.5% block GPTBot and only 5.1% publish an llms.txt file. Their content is their product, and many publishers do not want it used to train models without compensation. Regional newspapers that belong to the same group often share the same robots.txt, blocking up to 16 AI bots each.
At the other end, telecom, energy and hosting companies block almost no one and lead llms.txt adoption. They want AI assistants to know them well and recommend them.
Blocking AI vs inviting AI, by sector
% of websites in each sector
View the data as a table
| Sector | Sites | Blocks any AI bot | GPTBot | ClaudeBot | Google-Extended | llms.txt |
|---|---|---|---|---|---|---|
| Media | 59 | 57.6% | 47.5% | 47.5% | 8.5% | 5.1% |
| Health | 13 | 30.8% | 23.1% | 15.4% | 23.1% | 15.4% |
| Portals and classifieds | 14 | 28.6% | 14.3% | 21.4% | 14.3% | 7.1% |
| Ecommerce and retail | 33 | 21.2% | 12.1% | 12.1% | 12.1% | 18.2% |
| Universities | 37 | 18.9% | 8.1% | 16.2% | 8.1% | 10.8% |
| Travel and transport | 24 | 12.5% | 0% | 0% | 4.2% | 29.2% |
| Public sector | 48 | 12.5% | 12.5% | 10.4% | 8.3% | 2.1% |
| Banking and insurance | 28 | 7.1% | 7.1% | 3.6% | 0% | 32.1% |
| Tech and hosting | 16 | 6.2% | 0% | 0% | 0% | 43.8% |
| Telecom and energy | 22 | 0% | 0% | 0% | 0% | 40.9% |
| Automotive | 8 | 0% | 0% | 0% | 0% | 25% |
| Consumer brands | 12 | 0% | 0% | 0% | 0% | 0% |
Blocking training is not the same as blocking search
OpenAI, Anthropic and Perplexity use different crawlers to train their models and to search the web in real time when a user asks a question. Blocking the search crawler means not being cited in ChatGPT, Claude or Perplexity answers.
- OpenAI: GPTBot (training) is blocked by 15.3%, OAI-SearchBot (search) by 10.2%.
- Anthropic: ClaudeBot (training) is blocked by 15.6%, Claude-SearchBot (search) by 8.9%.
- Perplexity: PerplexityBot (search) is blocked by 11.5%.
Only 5.1% of the websites block OpenAI’s training crawler while allowing its search crawler, and just 1.3% block every training crawler we measured. Most websites either leave everything open or close everything.
The Google-Extended paradox
Google-Extended controls whether Google can use a site’s content to train and ground Gemini. It is blocked half as often as GPTBot (7% vs 15.3%). Blocking it does not affect rankings in Google Search, but many site owners either do not know that or prefer not to take the risk.
The public sector filters too
12.5% of the public sector websites in the sample block at least one AI bot. Some city councils use an allowlist: they let Googlebot and a few known crawlers in and block everything else, AI bots included.
What should your website do?
It depends on your business model. For most websites that depend on traffic and visibility, we recommend a selective strategy:
- Allow AI search and user bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User) so you can be cited in AI answers.
- Decide about training bots. If your content is your product, block GPTBot, ClaudeBot, Google-Extended, CCBot and similar crawlers.
- Check your CDN. Cloudflare and other providers offer AI bot blocking that can contradict your robots.txt.
- Publish an llms.txt file with a summary of your site and links to your key pages.
# AI search and user requests: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /
# AI model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /Check your own site in seconds with our free AI Bot Checker, or build a selective file with the robots.txt generator.
Methodology
Sample. 344 leading Spanish websites selected by SEOOn.page across 12 sectors. It is not a random sample of all Spanish websites: it represents the most visible players in each sector.
Valid sample: 314 websites. We excluded 20 websites whose robots.txt we could not read (bot protection or server errors) and 10 that did not respond. 19 websites have no robots.txt and count as allowing everything.
Interpretation. We evaluated every robots.txt following RFC 9309, the same logic Google uses: each bot follows its own group or, if it has none, the default group, and the most specific rule wins. A bot counts as fully blocked when it cannot reach any typical content page, even if the home page is allowed.
Bots measured. GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, Claude-User and anthropic-ai (Anthropic); PerplexityBot and Perplexity-User; Google-Extended; Applebot-Extended; CCBot; Bytespider; meta-externalagent; Amazonbot; cohere-ai; MistralAI-User and DuckAssistBot. Googlebot is included as a baseline.
What robots.txt does not show. We also requested each home page identifying ourselves as GPTBot and as ClaudeBot. 18.4% of the 304 comparable websites rejected those requests while serving a normal browser. We do not use this as a headline figure: bot protection systems also reject anyone who claims to be a known bot without coming from its official IP addresses, and we cannot tell the two cases apart.
Limitations. robots.txt is a public signal, not a guarantee: not every bot respects it, and some websites block bots at the server or CDN level. The data reflects October 14, 2026. We will repeat the study every quarter and publish an international edition.
Open data
We publish the full dataset, website by website, under a CC BY 4.0 license. You can use it freely with attribution and a link to this page. Download the CSV.
How to cite: SEOOn.page (2026). AI Bots and Spain’s Top Websites (2026). https://seoon.page/research/ai-bots-spain-top-websites-2026/
Journalists who need data for a specific sector can contact us.