Coding agents
Claude-Code is a bot operated by Anthropic that pulls documentation and examples while an AI writes code. Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-file edits,…. Whether it honours robots.txt is undocumented, and a request claiming to be Claude-Code can be checked against a published IP range file.
Request says
"Claude-Code"
Checked against
Anthropic's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Claude-CodeToken only: match on the substring. |
|---|---|
| Operator | Anthropic |
| Purpose | Coding agents |
| Respects robots.txt | No policy |
| Verification method | IP rangesPublished IP range file Runs on a developer machine or CI runner, so the source IP is usually the developer, not Anthropic. The published prefix file will not match most of this traffic. |
| Crawl pattern | Not published |
| Robots.txt tokens | Claude-Code |
| Operator documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-file edits,…
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Claude-Code
Disallow: /User-agent: Claude-Code
Allow: /User-agent: Claude-Code
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Claude-Code mainly affects developers reading your documentation through an AI tool. Most sites want that traffic.
Build a file for every crawlerFAQ
Claude-Code is a bot operated by Anthropic that pulls documentation and examples while an AI writes code. Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-file edits,…. Whether it honours robots.txt is undocumented, and a request claiming to be Claude-Code can be checked against a published IP range file.
Not documented. No operator statement says whether Claude-Code reads robots.txt, so add the rule but verify in your logs that the requests actually stop.
Runs on a developer machine or CI runner, so the source IP is usually the developer, not Anthropic. The published prefix file will not match most of this traffic. The reference is published at https://claude.com/crawling/bots.json.
Add the Claude-Code token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Claude-Code mainly affects developers reading your documentation through an AI tool. Most sites want that traffic.