Training crawlers
Meta-ExternalAgent is a bot operated by Meta that collects pages used to train AI models. Meta-ExternalAgent is a web crawler used by Meta to download training data for its AI models and improve its products by indexing content directly. It honours robots.txt, and a request claiming to be Meta-ExternalAgent can be checked against the operator's announced ASN.
Request says
"Meta-ExternalAgent"
Checked against
Meta's ASN
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Meta-ExternalAgentToken only: match on the substring. |
|---|---|
| Operator | Meta |
| Purpose | Training crawlers |
| Respects robots.txt | Honourssource |
| Verification method | Operator ASNOperator ASN check Meta does not publish a per-bot range file. The practical check is whether the source IP is announced by AS32934 (Facebook), which you can query from a routing registry. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
| Crawl pattern | Not published |
| Robots.txt tokens | Meta-ExternalAgentmeta-externalagent |
| Operator documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
Meta-ExternalAgent is a web crawler used by Meta to download training data for its AI models and improve its products by indexing content directly.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Disallow: /User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Allow: /User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Meta-ExternalAgent keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
Meta-ExternalAgent is a bot operated by Meta that collects pages used to train AI models. Meta-ExternalAgent is a web crawler used by Meta to download training data for its AI models and improve its products by indexing content directly. It honours robots.txt, and a request claiming to be Meta-ExternalAgent can be checked against the operator's announced ASN.
Yes. The operator documents that Meta-ExternalAgent reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Meta does not publish a per-bot range file. The practical check is whether the source IP is announced by AS32934 (Facebook), which you can query from a routing registry. The reference is published at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/.
Add these user-agent tokens (Meta-ExternalAgent, meta-externalagent) to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Meta-ExternalAgent keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.