Training crawlers
cohere-ai is a bot operated by Cohere that collects pages used to train AI models. Retrieves data based on user prompts. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Request says
"cohere-ai"
Checked against
Nothing published
Live check
No operator check exists for cohere-ai. This still shows what the address really is.
Reference
| User agent | cohere-aiToken only: match on the substring. |
|---|---|
| Operator | Cohere |
| Purpose | Retrieves data to provide responses to user-initiated prompts. |
| Respects robots.txt | No policy |
| Verification method | Claim onlyNone published, user agent only Cohere publishes no IP range list for this token. The separate cohere-training-data-crawler token is the one aimed at model training. |
| Crawl pattern | Takes action based on user prompts. |
| Robots.txt tokens | cohere-ai |
| Operator documentation | None published |
Retrieves data based on user prompts.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: cohere-ai
Disallow: /User-agent: cohere-ai
Allow: /User-agent: cohere-ai
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking cohere-ai keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
cohere-ai is a bot operated by Cohere that collects pages used to train AI models. Retrieves data based on user prompts. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Not documented. No operator statement says whether cohere-ai reads robots.txt, so add the rule but verify in your logs that the requests actually stop.
You cannot, at least not from the request itself. Cohere publishes no IP range list for this token. The separate cohere-training-data-crawler token is the one aimed at model training. Compare the source IP against hosting and datacenter ranges instead, and rate limit anything that behaves like a crawler.
Add the cohere-ai token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking cohere-ai keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.