SEO glossary

X-Robots-Tag

An HTTP response header that gives indexing instructions such as noindex or nofollow, like the meta robots tag but in the headers.

It is the only way to control indexing for non-HTML files such as PDFs and images. A forgotten X-Robots-Tag: noindex can silently remove pages from search results.

Check your headers with the HTTP Header Checker.

Related tools

XML Sitemap ValidatorFind and validate the XML sitemap of any website: XML errors, URL count, sitemap indexes, lastmod dates, duplicates and robots.txt declaration.HTTP Header CheckerSee all the HTTP response headers of any URL, with checks for indexing (X-Robots-Tag), compression, caching and security headers.Redirect CheckerFollow every redirect of a URL: see each hop, its status code (301, 302, 307, 308), redirect chains, loops and the final destination.

Learn more