Which Google range file should I use?

googlebot.json covers Search crawling. special-crawlers.json covers GoogleOther and the AdsBot family. user-triggered-fetchers.json and user-triggered-fetchers-google.json cover fetches a person asked for, including the ones behind Gemini and NotebookLM. Picking the wrong file is the most common reason a legitimate crawler gets blocked.

Does blocking Google-Extended block Googlebot?

No. Google-Extended is a robots.txt token only, never a user agent. It controls whether already-crawled pages are used for Gemini training and grounding, and it has no effect on Search crawling or ranking. Blocking Googlebot is the thing that removes you from Search.

How often do the published ranges change?

Google refreshes the JSON files without notice, so a copy pasted into a firewall rule goes stale. Fetch them on a schedule, or check requests against a service that does it for you.

What does fake Googlebot traffic usually look like?

A Googlebot user-agent arriving from a residential proxy pool, a cheap VPS provider or a datacenter ASN that is not Google. It normally ignores crawl-delay, requests pages in an order no real crawler would, and hammers endpoints that are expensive to render.

IPScanner

Free check

Verify Googlebot

A request is genuinely Googlebot only if the source IP passes a forward-confirmed reverse DNS check or appears in one of Google's published range files. Run a reverse lookup on the IP: it must return a hostname ending in googlebot.com or google.com, and a forward lookup of that hostname must return the same IP. The user-agent string proves nothing on its own, because anyone can send it.

Check an IP from your access log

Paste the address that sent the Googlebot user agent. The lookup returns the ASN and the network it belongs to, which is the fastest way to spot a fake: real Googlebot traffic comes from Google, not from a hosting reseller or a residential proxy pool.

Command line

The forward-confirmed reverse DNS check

Two lookups, one answer

# 1. reverse lookup on the address from your log
host 66.249.66.1
# -> 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2. forward lookup on the hostname you just got back
host crawl-66-249-66-1.googlebot.com
# -> crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# Genuine only if step 2 returns the exact address from step 1
# and the hostname ends in googlebot.com or google.com.

Replace the address with the one from your own log. A mismatch, or a hostname on any other domain, means the request was not Googlebot.

Do not skip step two. A PTR record can be set by whoever controls the address, so a googlebot.com hostname on its own is not proof. Only the forward lookup ties it back to Google’s zone.

Or match against the published range files

googlebot.jsonSearch crawling
special-crawlers.jsonGoogleOther and the AdsBot family
user-triggered-fetchers.jsonFetches a person asked for

Questions

Anyone can set a PTR record on an address they control, including one that ends in googlebot.com. The forward lookup is what closes that hole: only Google can make crawl-66-249-66-1.googlebot.com resolve back to the address that hit you.

Same problem, every other crawler

Googlebot is the easy case, because Google publishes both a DNS convention and range files. Most AI crawlers publish neither. The directory lists what each one actually offers.

AI crawler directory Googlebot reference GPTBot ClaudeBot PerplexityBot Bingbot Agentscan Datacenter IP detection