AI search crawlers
DuckDuckBot is a bot operated by DuckDuckGo that indexes pages so an AI answer engine can cite them. DuckDuckBot crawls for DuckDuckGo's own index. It honours robots.txt, and a request claiming to be DuckDuckBot can be checked against a published IP range file.
Request says
"DuckDuckBot"
Checked against
DuckDuckGo's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | DuckDuckBotToken only: match on the substring. |
|---|---|
| Operator | DuckDuckGo |
| Purpose | Search indexing |
| Respects robots.txt | Honourssource |
| Verification method | IP rangesPublished IP range file DuckDuckGo publishes the crawler addresses on its help pages. https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ |
| Crawl pattern | Light, most results come from partner indexes. |
| Robots.txt tokens | DuckDuckBot |
| Operator documentation | https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ |
DuckDuckBot crawls for DuckDuckGo's own index. It is distinct from DuckAssistBot, which fetches pages to answer questions in DuckDuckGo's AI assistant.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: DuckDuckBot
Disallow: /User-agent: DuckDuckBot
Allow: /User-agent: DuckDuckBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking DuckDuckBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
DuckDuckBot is a bot operated by DuckDuckGo that indexes pages so an AI answer engine can cite them. DuckDuckBot crawls for DuckDuckGo's own index. It honours robots.txt, and a request claiming to be DuckDuckBot can be checked against a published IP range file.
Yes. The operator documents that DuckDuckBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
DuckDuckGo publishes the crawler addresses on its help pages. The reference is published at https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/.
Add the DuckDuckBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking DuckDuckBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.