SEO glossary

Googlebot

Google’s main web crawler, which discovers and fetches pages for Google Search.

Googlebot crawls mainly with a smartphone user agent, following mobile-first indexing. Blocking it in robots.txt stops crawling but does not guarantee removal from the index: use noindex for that.

You can verify that a request really comes from Googlebot with a reverse DNS lookup or Google’s published IP ranges.