AI search crawlers
meta-webindexer is a bot operated by Meta that indexes pages so an AI answer engine can cite them. As per their documentation, "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. Whether it honours robots.txt is undocumented, and a request claiming to be meta-webindexer can be checked against the operator's announced ASN.
Request says
"meta-webindexer"
Checked against
Meta's ASN
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | meta-webindexerToken only: match on the substring. |
|---|---|
| Operator | Meta |
| Purpose | AI search crawlers |
| Respects robots.txt | No policy |
| Verification method | Operator ASNOperator ASN check Check the source IP against AS32934. Meta publishes no per-bot range file. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
| Crawl pattern | Unhinged, more than 1 per second. |
| Robots.txt tokens | meta-webindexer |
| Operator documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
As per their documentation, "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses."
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: meta-webindexer
Disallow: /User-agent: meta-webindexer
Allow: /User-agent: meta-webindexer
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking meta-webindexer takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
meta-webindexer is a bot operated by Meta that indexes pages so an AI answer engine can cite them. As per their documentation, "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. Whether it honours robots.txt is undocumented, and a request claiming to be meta-webindexer can be checked against the operator's announced ASN.
Not documented. No operator statement says whether meta-webindexer reads robots.txt, so add the rule but verify in your logs that the requests actually stop.
Check the source IP against AS32934. Meta publishes no per-bot range file. The reference is published at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/.
Add the meta-webindexer token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking meta-webindexer takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.