SEO glossary

Robots.txt

A text file at the root of a website that tells crawlers which URLs they may or may not crawl.

robots.txt rules are grouped by user agent, and the most specific matching rule wins. The format is standardized in RFC 9309.

robots.txt controls crawling, not indexing. Test any URL with our robots.txt tester.

Related tools

Domain LookupLook up any domain: registration and expiry dates, registrar, name servers, DNS records (A, AAAA, MX, NS, TXT), hosting network and SPF and DMARC.XML Sitemap ValidatorFind and validate the XML sitemap of any website: XML errors, URL count, sitemap indexes, lastmod dates, duplicates and robots.txt declaration.HTTP Header CheckerSee all the HTTP response headers of any URL, with checks for indexing (X-Robots-Tag), compression, caching and security headers.

Learn more