Training crawlers
Applebot-Extended is a bot operated by Apple that collects pages used to train AI models. Apple has a secondary user agent, Applebot-Extended ... The name never appears in a User-Agent header: it is a robots.txt directive only, so there is nothing to verify at request time.
robots.txt line
User-agent: Applebot-Extended
Checked against
Nothing: never sent as a user agent
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Applebot-ExtendedToken only: match on the substring. |
|---|---|
| Operator | Apple |
| Purpose | Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others. |
| Respects robots.txt | Honours |
| Verification method | Token onlyNot applicable, robots.txt token only Applebot-Extended is never sent as a User-Agent. It is a robots.txt token that tells Apple not to use already-crawled pages for Apple Intelligence training, while Applebot keeps crawling for Siri and Spotlight. |
| Crawl pattern | Not published |
| Robots.txt tokens | Applebot-Extended |
| Operator documentation | https://support.apple.com/en-us/119829 |
Apple has a secondary user agent, Applebot-Extended ... [that is] used to train Apple's foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: Applebot-Extended
Disallow: /User-agent: Applebot-Extended
Allow: /User-agent: Applebot-Extended
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking Applebot-Extended keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
Applebot-Extended is a bot operated by Apple that collects pages used to train AI models. Apple has a secondary user agent, Applebot-Extended ... The name never appears in a User-Agent header: it is a robots.txt directive only, so there is nothing to verify at request time.
Yes. The operator documents that Applebot-Extended reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Applebot-Extended is never sent as a User-Agent. It is a robots.txt token that tells Apple not to use already-crawled pages for Apple Intelligence training, while Applebot keeps crawling for Siri and Spotlight.
Add the Applebot-Extended token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking Applebot-Extended keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.