Training crawlers
ClaudeBot is a bot operated by Anthropic that collects pages used to train AI models. Scrapes data to train LLMs and AI products offered by Anthropic. It honours robots.txt, and a request claiming to be ClaudeBot can be checked against a published IP range file.
Request says
"ClaudeBot/1.0"
Checked against
Anthropic's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
|---|---|
| Operator | Anthropic |
| Purpose | Scrapes data to train Anthropic's AI products. |
| Respects robots.txt | Honourssource |
| Verification method | IP rangesPublished IP range file Anthropic publishes a single prefix file covering ClaudeBot, Claude-User and Claude-SearchBot. A request claiming to be ClaudeBot from outside those prefixes is a forgery. |
| Crawl pattern | Not published |
| Robots.txt tokens | ClaudeBot |
| Operator documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
Scrapes data to train LLMs and AI products offered by Anthropic.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: ClaudeBot
Disallow: /User-agent: ClaudeBot
Allow: /User-agent: ClaudeBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking ClaudeBot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
ClaudeBot is a bot operated by Anthropic that collects pages used to train AI models. Scrapes data to train LLMs and AI products offered by Anthropic. It honours robots.txt, and a request claiming to be ClaudeBot can be checked against a published IP range file.
Yes. The operator documents that ClaudeBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Anthropic publishes a single prefix file covering ClaudeBot, Claude-User and Claude-SearchBot. A request claiming to be ClaudeBot from outside those prefixes is a forgery. The reference is published at https://claude.com/crawling/bots.json.
Add the ClaudeBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking ClaudeBot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.