Assistant fetchers
Meta-ExternalFetcher is a bot operated by Meta that fetches single pages on demand for an AI assistant. Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. It is documented as not following robots.txt, and a request claiming to be Meta-ExternalFetcher can be checked against the operator's announced ASN.
Request says
"Meta-ExternalFetcher"
Checked against
Meta's ASN
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Meta-ExternalFetcherToken only: match on the substring. |
|---|---|
| Operator | Meta |
| Purpose | Assistant fetchers |
| Respects robots.txt | Ignoressource |
| Verification method | Operator ASNOperator ASN check Same AS32934 check as the other Meta agents. This one fetches a single URL a user shared with a Meta AI assistant. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
| Crawl pattern | Not published |
| Robots.txt tokens | Meta-ExternalFetchermeta-externalfetcher |
| Operator documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links.
robots.txt
Meta-ExternalFetcher is documented as ignoring robots.txt. Enforce a block at your CDN too.
User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Disallow: /User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Allow: /User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Meta-ExternalFetcher means a person who asks about one of your URLs gets an error instead of a summary. That is a support and reputation trade-off, not a training one.
Build a file for every crawlerFAQ
Meta-ExternalFetcher is a bot operated by Meta that fetches single pages on demand for an AI assistant. Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. It is documented as not following robots.txt, and a request claiming to be Meta-ExternalFetcher can be checked against the operator's announced ASN.
No. Meta-ExternalFetcher is documented as not following robots.txt, so a Disallow line will not stop it. Blocking needs to happen at the edge, by user agent or by source address.
Same AS32934 check as the other Meta agents. This one fetches a single URL a user shared with a Meta AI assistant. The reference is published at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/.
Add these user-agent tokens (Meta-ExternalFetcher, meta-externalfetcher) to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Meta-ExternalFetcher means a person who asks about one of your URLs gets an error instead of a summary. That is a support and reputation trade-off, not a training one.