AI search crawlers
Amazonbot is a bot operated by Amazon that indexes pages so an AI answer engine can cite them. Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses. It honours robots.txt, and a request claiming to be Amazonbot can be checked against a forward-confirmed reverse DNS lookup.
Request says
"Amazonbot/0.1"
Checked against
Amazon's reverse DNS
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) |
|---|---|
| Operator | Amazon |
| Purpose | Service improvement and enabling answers for Alexa users. |
| Respects robots.txt | Honours |
| Verification method | Reverse DNSReverse DNS, forward confirmed Verify with a reverse lookup: the PTR record resolves under the Amazon crawler domain, and a forward lookup of that hostname must return the original IP. |
| Crawl pattern | Not published |
| Robots.txt tokens | Amazonbot |
| Operator documentation | https://developer.amazon.com/amazonbot |
Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Amazonbot
Disallow: /User-agent: Amazonbot
Allow: /User-agent: Amazonbot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Amazonbot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
Amazonbot is a bot operated by Amazon that indexes pages so an AI answer engine can cite them. Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses. It honours robots.txt, and a request claiming to be Amazonbot can be checked against a forward-confirmed reverse DNS lookup.
Yes. The operator documents that Amazonbot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Verify with a reverse lookup: the PTR record resolves under the Amazon crawler domain, and a forward lookup of that hostname must return the original IP. The reference is published at https://developer.amazon.com/amazonbot.
Add the Amazonbot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Amazonbot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.