Training crawlers
Google-Extended is a bot operated by Google that collects pages used to train AI models. Used to train Gemini and Vertex AI generative APIs. The name never appears in a User-Agent header: it is a robots.txt directive only, so there is nothing to verify at request time.
robots.txt line
User-agent: Google-Extended
Checked against
Nothing: never sent as a user agent
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Google-ExtendedToken only: match on the substring. |
|---|---|
| Operator | |
| Purpose | LLM training. |
| Respects robots.txt | Honourssource |
| Verification method | Token onlyNot applicable, robots.txt token only Google-Extended never appears in a User-Agent header. It is a robots.txt token only, used to opt out of Gemini training and grounding while leaving Google Search crawling untouched. There is nothing to IP-verify. https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers |
| Crawl pattern | Not published |
| Robots.txt tokens | Google-Extended |
| Operator documentation | https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers |
Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Google-Extended
Disallow: /User-agent: Google-Extended
Allow: /User-agent: Google-Extended
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Google-Extended keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
Google-Extended is a bot operated by Google that collects pages used to train AI models. Used to train Gemini and Vertex AI generative APIs. The name never appears in a User-Agent header: it is a robots.txt directive only, so there is nothing to verify at request time.
Yes. The operator documents that Google-Extended reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Google-Extended never appears in a User-Agent header. It is a robots.txt token only, used to opt out of Gemini training and grounding while leaving Google Search crawling untouched. There is nothing to IP-verify.
Add the Google-Extended token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Google-Extended keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.