How to Verify Googlebot (and Block Bots Faking It)
Anyone can send a Googlebot User-Agent. Here's how to verify real Googlebot with reverse DNS, and how to tell verified bots, AI crawlers and malicious automation apart.
"Verify Googlebot" is one of the most searched bot questions for a reason: a huge share of unwanted traffic arrives wearing a Googlebot costume. The User-Agent is trivial to fake, so treating it as proof is how scrapers get the same free pass as the real crawler.
The User-Agent proves nothing
Googlebot/2.1 is just a header. A scraper, a headless browser, or a credential-stuffing
script can send the exact same string. If your allow rule is "User-Agent contains Googlebot,"
you have effectively published an open door with a sign that says do not enter.
The real check: forward-confirmed reverse DNS
Google publishes the method, and it has three steps:
- Reverse lookup the IP. Take the connecting IP and read its PTR record. Real Googlebot
resolves to a hostname under
googlebot.comorgoogle.com. - Confirm the domain. The hostname must end in one of those official domains. A datacenter IP claiming to be Googlebot usually has a PTR that does not.
- Forward lookup back. Resolve that hostname to an IP and confirm it matches the original. This closes the loop so a faked PTR record alone cannot pass.
A "Googlebot" request that fails any step is not Googlebot. The same pattern verifies Bingbot, and the other major crawlers publish their own ranges or DNS conventions.
It is not just Googlebot anymore
The harder problem in 2026 is everything in between human and obvious bot:
- AI crawlers like GPTBot, ClaudeBot and PerplexityBot, which you may want to allow, monetise, or block by name.
- Headless automation (Playwright, Puppeteer, HeadlessChrome) that impersonates a real browser convincingly.
- Spoofed good bots riding fake User-Agents from datacenter ranges.
Reverse DNS verifies the good bots. It does not, on its own, classify the rest.
Stacking signals to classify the rest
Verifying Googlebot is one input. To decide on everything else, combine it with:
- IP origin: datacenter, VPN, proxy or clean residential. Headless traffic from a datacenter range is the canonical malicious automation. Check an IP's reputation and origin first.
- Headless and automation tells:
webdriver, missing browser surfaces, and User-Agent markers like HeadlessChrome. - JA4 TLS fingerprint: reflects how the client actually speaks TLS, so it survives a spoofed User-Agent and separates a real Chrome from an impostor.
- Header consistency: real browsers send Accept, Accept-Language and Accept-Encoding together; many bots do not.
Allow the real ones, decide on the rest
| Request | Verification | Action |
|---|---|---|
| Googlebot, reverse DNS confirmed | Verified | Allow |
| AI crawler, identified by name | Known | Allow, monetise or block per policy |
| Headless from a datacenter | No good identity | Block |
| Spoofed Googlebot, reverse DNS fails | Failed | Block |
That is the model Agentscan runs in one request: it forward-confirms the verified allowlist by reverse DNS, identifies AI crawlers by name, and fuses IP origin with headless tells and JA4 to classify every request as human, known bot, AI agent or malicious automation. Verdicts are cached, so repeat lookups return in under 50ms.
Bottom line
Never trust a Googlebot User-Agent. Verify it with forward-confirmed reverse DNS, allow the crawlers you actually want, and stack IP origin, headless tells and JA4 to decide on everything else. That way you protect SEO and block the fakes at the same time.
FAQ