Robots.txt: The Complete Guide for SEO (2026)

Robots.txt: The Complete Guide for SEO (2026)

Written by

Mauricio Rodriguez

in

robots.txt is a plain text file at the root of your website, at yourdomain.com/robots.txt, that tells crawlers which parts of the site they may visit. It is one of the oldest web standards, formalized as RFC 9309 in 2022, and it is now also the main way to control AI crawlers. A small mistake in this file can hide an entire site from Google, so it is worth getting right.

What robots.txt does and does not do

  • It controls crawling: which URLs a bot may request.
  • It does not control indexing: a blocked URL can still appear in Google if other pages link to it, usually without a description. To keep a page out of search results, use a noindex tag and let it be crawled.
  • It is a public request, not security: anyone can read your robots.txt, and bad bots ignore it. Never use it to hide private content.

The syntax

A robots.txt file is made of groups. Each group starts with one or more User-agent lines and contains Allow and Disallow rules. A Sitemap line can go anywhere.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml
  • User-agent: the bot the group applies to. * means every bot without a group of its own.
  • Disallow: a path prefix the bot should not crawl. An empty Disallow allows everything.
  • Allow: an exception inside a disallowed path.
  • * in a path matches any characters, and $ marks the end of the URL.
  • Paths are case sensitive; user agent names are not.

How Google decides which rule applies

  • A bot follows only the most specific group that names it. If GPTBot has its own group, it ignores the * group completely.
  • Inside that group, the longest matching rule wins, regardless of order.
  • If an Allow and a Disallow rule have the same length, Allow wins.
  • If no rule matches, the URL is allowed.

Test any URL against your rules, and see which line decides, with our robots.txt tester and generator.

Examples

A typical WordPress site

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

An online store

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?filter_

Sitemap: https://example.com/sitemap_index.xml

Blocking sorting and filter parameters saves crawl budget on large catalogs, but make sure your category pages themselves stay crawlable.

Robots.txt and AI bots

AI companies use separate crawlers for training models and for AI search. You can block one and allow the other. Read should you block AI bots? for ready-made rules, and check what your file allows today with the AI Bot Checker.

Common mistakes

  • Leaving Disallow: / from a staging site after launch. It blocks the whole site.
  • Blocking CSS and JavaScript, which Google needs to render pages.
  • Blocking pages you want to noindex: Google cannot see the noindex tag if it cannot crawl the page.
  • Adding a group for one bot and forgetting that it no longer follows the general rules.
  • Serving robots.txt with an error. If the file returns a 5xx error for a long time, Google may stop crawling the site. A 404 is fine: it means everything is allowed.

Frequently asked questions

Do I need a robots.txt file?

No. Without one, every crawler can access everything, which is fine for many small sites. It becomes useful when you need to keep bots out of some sections or control AI crawlers.

How long does Google take to see changes?

Google usually refreshes its cached copy of robots.txt within 24 hours.

Where do I edit robots.txt in WordPress?

WordPress generates a virtual robots.txt. SEO plugins such as Rank Math or Yoast let you edit it, unless a physical robots.txt file exists in the root folder, which always takes priority.

More posts