AI search crawlers
Bingbot is a bot operated by Microsoft that indexes pages so an AI answer engine can cite them. Bingbot indexes pages for Bing Search. It honours robots.txt, and a request claiming to be Bingbot can be checked against a published IP range file or a forward-confirmed reverse DNS lookup.
Request says
"bingbot/2.0"
Checked against
Microsoft's IP ranges and reverse DNS
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 |
|---|---|
| Operator | Microsoft |
| Purpose | Search indexing that also feeds Copilot answers |
| Respects robots.txt | Honourssource |
| Verification method | IP ranges + DNSPublished IP ranges or reverse DNS Reverse DNS must resolve to a search.msn.com host and forward-resolve back to the same IP. Microsoft also publishes the ranges as JSON. |
| Crawl pattern | Continuous, controllable via Bing Webmaster Tools crawl control. |
| Robots.txt tokens | Bingbot |
| Operator documentation | https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 |
Bingbot indexes pages for Bing Search. The same index grounds Microsoft Copilot answers, so a Bingbot block removes a site from Copilot citations as well as from Bing results.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Bingbot
Disallow: /User-agent: Bingbot
Allow: /User-agent: Bingbot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Bingbot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
Bingbot is a bot operated by Microsoft that indexes pages so an AI answer engine can cite them. Bingbot indexes pages for Bing Search. It honours robots.txt, and a request claiming to be Bingbot can be checked against a published IP range file or a forward-confirmed reverse DNS lookup.
Yes. The operator documents that Bingbot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Reverse DNS must resolve to a search.msn.com host and forward-resolve back to the same IP. Microsoft also publishes the ranges as JSON. The reference is published at https://www.bing.com/toolbox/bingbot.json.
Add the Bingbot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Bingbot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.