Training crawlers
FacebookBot is a bot operated by Meta/Facebook that collects pages used to train AI models. Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically. It honours robots.txt, and a request claiming to be FacebookBot can be checked against the operator's announced ASN.
Request says
"FacebookBot"
Checked against
Meta/Facebook's ASN
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | FacebookBotToken only: match on the substring. |
|---|---|
| Operator | Meta/Facebook |
| Purpose | Training language models |
| Respects robots.txt | Honourssource |
| Verification method | Operator ASNOperator ASN check Check the source IP against AS32934. Meta publishes no per-bot range file. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
| Crawl pattern | Up to 1 page per second |
| Robots.txt tokens | FacebookBot |
| Operator documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: FacebookBot
Disallow: /User-agent: FacebookBot
Allow: /User-agent: FacebookBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking FacebookBot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
FacebookBot is a bot operated by Meta/Facebook that collects pages used to train AI models. Officially used for training Meta "speech recognition technology," unknown if used to train Meta AI specifically. It honours robots.txt, and a request claiming to be FacebookBot can be checked against the operator's announced ASN.
Yes. The operator documents that FacebookBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Check the source IP against AS32934. Meta publishes no per-bot range file. The reference is published at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/.
Add the FacebookBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking FacebookBot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.