Training crawlers
Bytespider is a bot operated by ByteDance that collects pages used to train AI models. Downloads data to train LLMS, including ChatGPT competitors. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Request says
"Bytespider"
Checked against
Nothing published
Live check
No operator check exists for Bytespider. This still shows what the address really is.
Reference
| User agent | BytespiderToken only: match on the substring. |
|---|---|
| Operator | ByteDance |
| Purpose | LLM training. |
| Respects robots.txt | Ignores |
| Verification method | Claim onlyNone published, user agent only ByteDance publishes no IP ranges and no reverse-DNS convention for Bytespider, and it has been widely reported crawling aggressively from addresses that do not resolve back to ByteDance. Rate limiting is more reliable than a robots.txt rule. |
| Crawl pattern | Not published |
| Robots.txt tokens | Bytespider |
| Operator documentation | None published |
Downloads data to train LLMS, including ChatGPT competitors.
Bytespider is run by ByteDance, whose cloud, BytePlus, is one of the networks behind the Singapore traffic many sites see in their analytics. China and Singapore bot traffic separates the cloud bots from real visitors.
robots.txt
Bytespider is documented as ignoring robots.txt. Enforce a block at your CDN too.
User-agent: Bytespider
Disallow: /User-agent: Bytespider
Allow: /User-agent: Bytespider
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Bytespider keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
Bytespider is a bot operated by ByteDance that collects pages used to train AI models. Downloads data to train LLMS, including ChatGPT competitors. No IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine and a robots.txt rule alone will not stop a forgery.
Site-owner reports suggest Bytespider does not reliably follow robots.txt. Add the rule anyway, then check your logs; if requests continue, block at the edge by user agent or source address.
You cannot, at least not from the request itself. ByteDance publishes no IP ranges and no reverse-DNS convention for Bytespider, and it has been widely reported crawling aggressively from addresses that do not resolve back to ByteDance. Rate limiting is more reliable than a robots.txt rule. Compare the source IP against hosting and datacenter ranges instead, and rate limit anything that behaves like a crawler.
Add the Bytespider token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Bytespider keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.