Dataset builders
Diffbot is a bot that crawls pages into a dataset that may be reused or shared with third parties. Diffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Request says
"Diffbot"
Checked against
Nothing published
Live check
No operator check exists for Diffbot. This still shows what the address really is.
Reference
| User agent | DiffbotToken only: match on the substring. |
|---|---|
| Operator | Diffbot |
| Purpose | Dataset builders |
| Respects robots.txt | No policy |
| Verification method | Claim onlyNone published, user agent only Whether it obeys robots.txt is set per customer, so two Diffbot crawls of the same site can behave differently. No published IP list. |
| Crawl pattern | Not published |
| Robots.txt tokens | Diffbot |
| Operator documentation | None published |
Diffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Diffbot
Disallow: /User-agent: Diffbot
Allow: /User-agent: Diffbot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Diffbot keeps your pages out of the datasets it builds from future crawls. It does not remove anything collected earlier, and how each operator uses or shares its data varies, so check their policy.
Build a file for every crawlerFAQ
Diffbot is a bot that crawls pages into a dataset that may be reused or shared with third parties. Diffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Not documented. No operator statement says whether Diffbot reads robots.txt, so add the rule but verify in your logs that the requests actually stop.
You cannot, at least not from the request itself. Whether it obeys robots.txt is set per customer, so two Diffbot crawls of the same site can behave differently. No published IP list. Compare the source IP against hosting and datacenter ranges instead, and rate limit anything that behaves like a crawler.
Add the Diffbot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Diffbot keeps your pages out of the datasets it builds from future crawls. It does not remove anything collected earlier, and how each operator uses or shares its data varies, so check their policy.