Training crawlers
AI2Bot is a bot operated by Ai2 that collects pages used to train AI models. Explores 'certain domains' to find web content. It is documented as honouring robots.txt, but no IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine.
Request says
"AI2Bot"
Checked against
Nothing published
Live check
No operator check exists for AI2Bot. This still shows what the address really is.
Reference
| User agent | AI2BotToken only: match on the substring. |
|---|---|
| Operator | Ai2 |
| Purpose | Content is used to train open language models. |
| Respects robots.txt | Honours |
| Verification method | Claim onlyNone published, user agent only Ai2 documents the crawler and an opt-out but publishes no address list. Its output feeds the open OLMo and Dolma datasets, so a block here removes the site from public research corpora as well as from a model. |
| Crawl pattern | Not published |
| Robots.txt tokens | AI2BotAi2Bot-Dolma |
| Operator documentation | https://allenai.org/crawler |
Explores 'certain domains' to find web content.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Disallow: /User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Allow: /User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking AI2Bot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
AI2Bot is a bot operated by Ai2 that collects pages used to train AI models. Explores 'certain domains' to find web content. It is documented as honouring robots.txt, but no IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine.
Yes. The operator documents that AI2Bot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
You cannot, at least not from the request itself. Ai2 documents the crawler and an opt-out but publishes no address list. Its output feeds the open OLMo and Dolma datasets, so a block here removes the site from public research corpora as well as from a model. Compare the source IP against hosting and datacenter ranges instead, and rate limit anything that behaves like a crawler.
Add these user-agent tokens (AI2Bot, Ai2Bot-Dolma) to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking AI2Bot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.