AI search crawlers
Claude-SearchBot is a bot operated by Anthropic that indexes pages so an AI answer engine can cite them. Claude-SearchBot navigates the web to improve search result quality for users. It honours robots.txt, and a request claiming to be Claude-SearchBot can be checked against a published IP range file.
Request says
"Claude-SearchBot/1.0"
Checked against
Anthropic's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com) |
|---|---|
| Operator | Anthropic |
| Purpose | Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. |
| Respects robots.txt | Honourssource |
| Verification method | IP rangesPublished IP range file Indexes pages so Claude can cite them in search answers. Same published prefix file as the other Anthropic agents. |
| Crawl pattern | Not published |
| Robots.txt tokens | Claude-SearchBot |
| Operator documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Claude-SearchBot
Disallow: /User-agent: Claude-SearchBot
Allow: /User-agent: Claude-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Claude-SearchBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
Claude-SearchBot is a bot operated by Anthropic that indexes pages so an AI answer engine can cite them. Claude-SearchBot navigates the web to improve search result quality for users. It honours robots.txt, and a request claiming to be Claude-SearchBot can be checked against a published IP range file.
Yes. The operator documents that Claude-SearchBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Indexes pages so Claude can cite them in search answers. Same published prefix file as the other Anthropic agents. The reference is published at https://claude.com/crawling/bots.json.
Add the Claude-SearchBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Claude-SearchBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.