Training crawlers
anthropic-ai is a bot operated by Anthropic that collects pages used to train AI models. Scrapes data to train LLMs and AI products offered by Anthropic. Whether it honours robots.txt is undocumented, and a request claiming to be anthropic-ai can be checked against a published IP range file.
Request says
"anthropic-ai"
Checked against
Anthropic's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | anthropic-aiToken only: match on the substring. |
|---|---|
| Operator | Anthropic |
| Purpose | Scrapes data to train Anthropic's AI products. |
| Respects robots.txt | No policy |
| Verification method | IP rangesPublished IP range file An older token widely copied into robots.txt files. Anthropic now identifies its crawlers as ClaudeBot, Claude-User and Claude-SearchBot, so keep this rule but do not rely on it alone. |
| Crawl pattern | Not published |
| Robots.txt tokens | anthropic-ai |
| Operator documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
Scrapes data to train LLMs and AI products offered by Anthropic.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: anthropic-ai
Disallow: /User-agent: anthropic-ai
Allow: /User-agent: anthropic-ai
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking anthropic-ai keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
anthropic-ai is a bot operated by Anthropic that collects pages used to train AI models. Scrapes data to train LLMs and AI products offered by Anthropic. Whether it honours robots.txt is undocumented, and a request claiming to be anthropic-ai can be checked against a published IP range file.
Not documented. No operator statement says whether anthropic-ai reads robots.txt, so add the rule but verify in your logs that the requests actually stop.
An older token widely copied into robots.txt files. Anthropic now identifies its crawlers as ClaudeBot, Claude-User and Claude-SearchBot, so keep this rule but do not rely on it alone. The reference is published at https://claude.com/crawling/bots.json.
Add the anthropic-ai token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking anthropic-ai keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.