Other automated crawlers
facebookexternalhit is a bot operated by Meta/Facebook that crawls the web automatically. Note that excluding FacebookExternalHit will block incorporating OpenGraph data when sharing in social media, including rich links in Apple's Messages app. It is documented as not following robots.txt, and a request claiming to be facebookexternalhit can be checked against the operator's announced ASN.
Request says
"facebookexternalhit"
Checked against
Meta/Facebook's ASN
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | facebookexternalhitToken only: match on the substring. |
|---|---|
| Operator | Meta/Facebook |
| Purpose | Ostensibly only for sharing, but likely used as an AI crawler as well |
| Respects robots.txt | Ignoressource |
| Verification method | Operator ASNOperator ASN check The link-preview scraper. It has existed far longer than the AI agents and is the most commonly spoofed Meta token, so check AS32934 before trusting it. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
| Crawl pattern | Not published |
| Robots.txt tokens | facebookexternalhit |
| Operator documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
Note that excluding FacebookExternalHit will block incorporating OpenGraph data when sharing in social media, including rich links in Apple's Messages app. According to Meta, its purpose is "to crawl the content of an app or website that was shared on one of Meta’s family of apps…". However, see discussions here and here for evidence to the contrary.
robots.txt
facebookexternalhit is documented as ignoring robots.txt. Enforce a block at your CDN too.
User-agent: facebookexternalhit
Disallow: /User-agent: facebookexternalhit
Allow: /User-agent: facebookexternalhit
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking facebookexternalhit stops this specific token. Check your logs afterwards, because operators that ignore robots.txt often come back under a different name.
Build a file for every crawlerFAQ
facebookexternalhit is a bot operated by Meta/Facebook that crawls the web automatically. Note that excluding FacebookExternalHit will block incorporating OpenGraph data when sharing in social media, including rich links in Apple's Messages app. It is documented as not following robots.txt, and a request claiming to be facebookexternalhit can be checked against the operator's announced ASN.
No. facebookexternalhit is documented as not following robots.txt, so a Disallow line will not stop it. Blocking needs to happen at the edge, by user agent or by source address.
The link-preview scraper. It has existed far longer than the AI agents and is the most commonly spoofed Meta token, so check AS32934 before trusting it. The reference is published at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/.
Add the facebookexternalhit token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking facebookexternalhit stops this specific token. Check your logs afterwards, because operators that ignore robots.txt often come back under a different name.