Autonomous agents
Crawl4AI is a bot operated by Crawl4AI (open source) that browses the web on its own to finish multi-step tasks. Crawl4AI is an open source crawler library that converts pages into markdown for LLM pipelines. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Request says
"Crawl4AI"
Checked against
Nothing published
Live check
No operator check exists for Crawl4AI. This still shows what the address really is.
Reference
| User agent | Crawl4AIToken only: match on the substring. |
|---|---|
| Operator | Crawl4AI (open source) |
| Purpose | Autonomous agents |
| Respects robots.txt | No policy |
| Verification method | Claim onlyNone published, user agent only Open source crawler library used to feed pages into LLM pipelines. Anyone can run it under any user agent, so this token is a courtesy rather than an identity. |
| Crawl pattern | Not published |
| Robots.txt tokens | Crawl4AI |
| Operator documentation | None published |
Crawl4AI is an open source crawler library that converts pages into markdown for LLM pipelines. The token is whatever the person running it left in the config.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Crawl4AI
Disallow: /User-agent: Crawl4AI
Allow: /User-agent: Crawl4AI
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Crawl4AI stops automated sessions that fill forms, click through flows and read pages the way a logged-in user would. Rate limits and challenges usually work better here than a robots.txt line.
Build a file for every crawlerFAQ
Crawl4AI is a bot operated by Crawl4AI (open source) that browses the web on its own to finish multi-step tasks. Crawl4AI is an open source crawler library that converts pages into markdown for LLM pipelines. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Not documented. No operator statement says whether Crawl4AI reads robots.txt, so add the rule but verify in your logs that the requests actually stop.
You cannot, at least not from the request itself. Open source crawler library used to feed pages into LLM pipelines. Anyone can run it under any user agent, so this token is a courtesy rather than an identity. Compare the source IP against hosting and datacenter ranges instead, and rate limit anything that behaves like a crawler.
Add the Crawl4AI token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Crawl4AI stops automated sessions that fill forms, click through flows and read pages the way a logged-in user would. Rate limits and challenges usually work better here than a robots.txt line.