AI search crawlers
Googlebot is a bot operated by Google that indexes pages so an AI answer engine can cite them. Googlebot is the crawler behind Google Search. It honours robots.txt, and a request claiming to be Googlebot can be checked against a published IP range file or a forward-confirmed reverse DNS lookup.
Request says
"Googlebot/2.1"
Checked against
Google's IP ranges and reverse DNS
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
|---|---|
| Operator | |
| Purpose | Search indexing that also feeds AI Overviews |
| Respects robots.txt | Honourssource |
| Verification method | IP ranges + DNSPublished IP ranges or reverse DNS Two options: reverse DNS on the source IP must return a googlebot.com or google.com host and a forward lookup of that host must return the same IP, or the IP must appear in googlebot.json. https://developers.google.com/static/search/apis/ipranges/googlebot.json |
| Crawl pattern | Continuous, rate adapted per site. |
| Robots.txt tokens | Googlebot |
| Operator documentation | https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot |
Googlebot is the crawler behind Google Search. It is not an AI training crawler, but pages it indexes can be summarised in AI Overviews and used for grounding in Gemini, so blocking Google-Extended without blocking Googlebot is the usual way to stay in Search while opting out of training.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Googlebot
Disallow: /User-agent: Googlebot
Allow: /User-agent: Googlebot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Googlebot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
Googlebot is a bot operated by Google that indexes pages so an AI answer engine can cite them. Googlebot is the crawler behind Google Search. It honours robots.txt, and a request claiming to be Googlebot can be checked against a published IP range file or a forward-confirmed reverse DNS lookup.
Yes. The operator documents that Googlebot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Two options: reverse DNS on the source IP must return a googlebot.com or google.com host and a forward lookup of that host must return the same IP, or the IP must appear in googlebot.json. The reference is published at https://developers.google.com/static/search/apis/ipranges/googlebot.json.
Add the Googlebot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Googlebot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.