SEO glossary

Crawling

The process in which search engine bots discover and download web pages by following links and reading sitemaps.

Crawling comes before indexing: a page that cannot be crawled, for example because robots.txt blocks it, cannot be properly indexed or ranked.

Help crawlers find your pages with a clean sitemap. Check yours with the XML Sitemap Validator.

Related tools

XML Sitemap ValidatorFind and validate the XML sitemap of any website: XML errors, URL count, sitemap indexes, lastmod dates, duplicates and robots.txt declaration.HTTP Header CheckerSee all the HTTP response headers of any URL, with checks for indexing (X-Robots-Tag), compression, caching and security headers.Redirect CheckerFollow every redirect of a URL: see each hop, its status code (301, 302, 307, 308), redirect chains, loops and the final destination.

Learn more