Robots.txt Tester and Generator

Test any URL against robots.txt

Load the robots.txt of any website (or paste your own draft), pick a user agent and type a URL path. The tester tells you instantly whether that bot is allowed to crawl the path and shows the exact line that decided it.

Since Google retired the robots.txt tester from Search Console, this is a quick way to check rules before you publish them.

How robots.txt rules are evaluated

• A bot looks for a group with its own name (for example User-agent: Googlebot). If there is none, it follows the User-agent: * group.
• Inside the group, the longest matching rule wins. Allow: /blog/ beats Disallow: / for URLs under /blog/.
• If an Allow and a Disallow rule are equally long, Allow wins.
• * matches any sequence of characters and $ marks the end of the URL.
• If no rule matches, the URL is allowed.

Generate a robots.txt with AI presets

The generator builds a clean file in three strategies:

• Selective (recommended for most sites): AI search and user bots are allowed, AI training bots are blocked.
• Open: every bot can crawl everything.
• Block all AI bots: AI search, user and training bots are blocked; classic search engines are still allowed.

Add your private paths (such as /wp-admin/ or /cart/) and your sitemap URL, then copy or download the file.

Common robots.txt mistakes

• Blocking CSS or JavaScript files that Google needs to render the page.
• Using robots.txt to hide pages from search results. It stops crawling, not indexing: use a noindex tag instead.
• Adding a specific group for a bot and forgetting that it no longer follows the * rules.
• Leaving Disallow: / from a staging site after launch.

Frequently asked questions