Crawl4AI
Crawl4AI is a bot operated by Crawl4AI (open source) that browses the web on its own to finish multi-step tasks. Crawl4AI is an open source crawler library that converts pages into markdown for LLM pipelines. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Crawl4AI at a glance
| User agent | Crawl4AIToken only. The operator has not published a full user-agent string, so match on the substring rather than an exact header. |
|---|---|
| Operator | Crawl4AI (open source) |
| Purpose | Autonomous agents |
| Respects robots.txt | Not documented |
| Verification method | None published, user agent only Open source crawler library used to feed pages into LLM pipelines. Anyone can run it under any user agent, so this token is a courtesy rather than an identity. https://github.com/unclecode/crawl4ai |
| Crawl pattern | Not published |
| Robots.txt tokens | Crawl4AI |
| Operator documentation | None published |
Crawl4AI is an open source crawler library that converts pages into markdown for LLM pipelines. The token is whatever the person running it left in the config.
Allow or block Crawl4AI
Paste one of these into the robots.txt file at the root of your domain. Rules are per token, so a block on one crawler leaves every other bot untouched.
User-agent: Crawl4AI
Disallow: /Blocks every path for this crawler only.
User-agent: Crawl4AI
Allow: /Explicit allow. Useful when a wildcard rule above it would otherwise catch this crawler.
User-agent: Crawl4AI
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own account, checkout and search URLs.
What blocking actually costs you
Blocking Crawl4AI stops automated sessions that fill forms, click through flows and read pages the way a logged-in user would. Rate limits and challenges usually work better here than a robots.txt line.
Block every crawler in the directory at onceIs that really Crawl4AI?
A user-agent header is a string the client chooses. Scrapers copy Crawl4AI precisely because site owners allow it. Open source crawler library used to feed pages into LLM pipelines. Anyone can run it under any user agent, so this token is a courtesy rather than an identity.
Paste the IP address from your access log below. You will see whether it belongs to a hosting provider, a residential proxy pool or a Tor exit, plus the ASN that announces it, which is what tells you whether the claim holds up.
Related crawlers
Identify crawlers automatically instead of by hand
Agentscan checks a request against published crawler ranges, reverse DNS and hosting data, then returns a verdict your edge can act on. One call per request, no range files to keep up to date.