AI crawler directory
Who runs each AI crawler, what it sends, and whether a request using its name can be proven.
Claim only
78%
119 of 153 crawler tokens publish no way to check a request.
From the ai.robots.txt list and operator docs, checked against what each operator publishes. Updated .
Collect pages for a model's training corpus. Blocking these is the usual AI opt-out.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| AI2BotAi2Bot-Dolma | Ai2 | Claim only | robots.txt:Honours | |
| anthropic-ai | Anthropic | IP ranges | robots.txt:No policy | |
| Applebot-Extended | Apple | Token only | robots.txt:Honours | |
| Bytespider | ByteDance | Claim only | robots.txt:Ignores | |
| ClaudeBot | Anthropic | IP ranges | robots.txt:Honours | |
| cohere-ai | Cohere | Claim only | robots.txt:No policy | |
| Cotoyogi | ROIS | Claim only | robots.txt:Honours | |
| DeepSeekBot | DeepSeek | Claim only | robots.txt:Ignores | |
| FacebookBot | Meta/Facebook | Operator ASN | robots.txt:Honours | |
| Factset_spyderbot | Factset | Claim only | robots.txt:No policy | |
| FriendlyCrawler | Unknown | Claim only | robots.txt:Honours | |
| Google-Extended | Token only | robots.txt:Honours | ||
| GPTBot | OpenAI | IP ranges | robots.txt:Honours | |
| ICC-Crawler | NICT | Claim only | robots.txt:Honours | |
| img2dataset | img2dataset | Claim only | robots.txt:No policy | |
| ISSCyberRiskCrawler | ISS-Corporate | Claim only | robots.txt:Ignores | |
| Linguee Bot | Linguee | Claim only | robots.txt:Ignores | |
| Meta-ExternalAgentmeta-externalagent | Meta | Operator ASN | robots.txt:Honours | |
| Scrapy | Zyte | Claim only | robots.txt:No policy | |
| Sidetrade indexer bot | Sidetrade | Claim only | robots.txt:No policy | |
| TikTokSpider | ByteDance | Claim only | robots.txt:No policy |
Build the index an AI answer engine cites from. Blocking one removes you from its answers.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| AddSearchBot | Unidentified | Claim only | robots.txt:No policy | |
| AIWebIndexAIWebIndex-Agent | Lyrenth (Aleksma Ai, Inc.) | IP ranges + DNS | robots.txt:Honours | |
| amazon-kendra | Amazon | Claim only | robots.txt:Honours | |
| Amazonbot | Amazon | Reverse DNS | robots.txt:Honours | |
| Amzn-SearchBot | Amazon | Reverse DNS | robots.txt:No policy | |
| Andibot | Andi | Claim only | robots.txt:No policy | |
| Anomura | Direqt | Claim only | robots.txt:Honours | |
| Applebot | Apple | IP ranges + DNS | robots.txt:Honours | |
| atlassian-bot | Atlassian | Claim only | robots.txt:Honours | |
| AzureAI-SearchBot | Microsoft | Claim only | robots.txt:No policy | |
| Bingbot | Microsoft | IP ranges + DNS | robots.txt:Honours | |
| Channel3Bot | Unidentified | Claim only | robots.txt:No policy | |
| Claude-SearchBot | Anthropic | IP ranges | robots.txt:Honours | |
| Cloudflare-AutoRAG | Cloudflare | Claim only | robots.txt:Honours | |
| DuckDuckBot | DuckDuckGo | IP ranges | robots.txt:Honours | |
| ExaSearchBot | Exa | Claim only | robots.txt:No policy | |
| Googlebot | IP ranges + DNS | robots.txt:Honours | ||
| GoogleOther | IP ranges + DNS | robots.txt:Honours | ||
| GoogleOther-Image | IP ranges + DNS | robots.txt:Honours | ||
| GoogleOther-Video | IP ranges + DNS | robots.txt:Honours | ||
| ImagesiftBot | ImageSift | Claim only | robots.txt:Honours | |
| LinkupBot | Linkup | Claim only | robots.txt:No policy | |
| meta-webindexer | Meta | Operator ASN | robots.txt:No policy | |
| OAI-SearchBot | OpenAI | IP ranges | robots.txt:Honours | |
| PerplexityBot | Perplexity | IP ranges | robots.txt:Honours | |
| PetalBot | Huawei | Claim only | robots.txt:Honours | |
| Timpibot | Timpi | Claim only | robots.txt:No policy | |
| YouBot | You | Claim only | robots.txt:Honours | |
| ZanistaBot | Unidentified | Claim only | robots.txt:No policy |
Fetch one page because a person asked an assistant about it. Some skip robots.txt.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute | Claim only | robots.txt:No policy | |
| amazon-QBusiness | Amazon Web Services | Claim only | robots.txt:No policy | |
| Amzn-User | Amazon | Reverse DNS | robots.txt:No policy | |
| bigsur.ai | Unidentified | Claim only | robots.txt:No policy | |
| ChatGPT-User | OpenAI | IP ranges | robots.txt:Honours | |
| Claude-User | Anthropic | IP ranges | robots.txt:Honours | |
| DuckAssistBot | Unidentified | IP ranges | robots.txt:Honours | |
| GeistHaus-PageFetcher | Unidentified | Claim only | robots.txt:No policy | |
| Gemini-Deep-Research | Claim only | robots.txt:No policy | ||
| Google-NotebookLMNotebookLM | IP ranges | robots.txt:No policy | ||
| GoogleAgent-URLContext | Claim only | robots.txt:No policy | ||
| kagi-fetcher | Unidentified | Claim only | robots.txt:No policy | |
| Kimi-User | Moonshot AI | Claim only | robots.txt:No policy | |
| KlaviyoAIBot | Klaviyo | Claim only | robots.txt:Honours | |
| LinerBot | Unidentified | Claim only | robots.txt:No policy | |
| Meta-ExternalFetchermeta-externalfetcher | Meta | Operator ASN | robots.txt:Ignores | |
| MistralAI-UserMistralAI-User/1.0 | Mistral | Claim only | robots.txt:No policy | |
| Perplexity-User | Perplexity | IP ranges | robots.txt:Ignores | |
| PhindBot | phind | Claim only | robots.txt:No policy | |
| Poggio-Citations | Unidentified | Claim only | robots.txt:No policy | |
| QualifiedBot | Qualified | Claim only | robots.txt:No policy | |
| SemrushBot-SWA | Semrush | Claim only | robots.txt:Honours | |
| Shap-User | Parallel | Claim only | robots.txt:No policy | |
| TongyiBot | Alibaba | Claim only | robots.txt:No policy | |
| UseAI | Unidentified | Claim only | robots.txt:No policy | |
| YiyanBot | Baidu that fetches web content for the yiyan | Claim only | robots.txt:No policy |
Drive a browser through multi-step tasks, much like a human session.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| AmazonBuyForMe | Amazon | Claim only | robots.txt:No policy | |
| Aranet-SearchBot | Unidentified | Claim only | robots.txt:No policy | |
| ChatGPT Agent | OpenAI | IP ranges | robots.txt:Honours | |
| Claude-Web | Anthropic | IP ranges | robots.txt:No policy | |
| Crawl4AI | Crawl4AI (open source) | Claim only | robots.txt:No policy | |
| Google-Agent | Claim only | robots.txt:Honours | ||
| Google-CloudVertexBotCloudVertexBot | IP ranges | robots.txt:Honours | ||
| GoogleAgent-Mariner | Claim only | robots.txt:No policy | ||
| iAskBotiaskspider, iaskspider/2.0 | iAsk | Claim only | robots.txt:No policy | |
| KunatoCrawler | Unidentified | Claim only | robots.txt:No policy | |
| Manus-User | Butterfly Effect, a company based in China | Claim only | robots.txt:No policy | |
| NagetBot | Unidentified | Claim only | robots.txt:No policy | |
| newsai | Unidentified | Claim only | robots.txt:No policy | |
| NovaAct | Amazon | Claim only | robots.txt:No policy | |
| Operator | OpenAI | IP ranges | robots.txt:No policy | |
| Reflectionbot | Reflection | Claim only | robots.txt:No policy | |
| TwinAgent | Unidentified | Claim only | robots.txt:No policy | |
| WRTNBot | Wrtn Technologies | Claim only | robots.txt:No policy |
Pull docs and examples while an AI writes code, often from a developer machine.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| Claude-Code | Anthropic | IP ranges | robots.txt:No policy | |
| Code | GitHub Copilot | Claim only | robots.txt:No policy | |
| Cursor | Cursor | Claim only | robots.txt:No policy | |
| Devin | Devin AI | Claim only | robots.txt:Honours | |
| Google-Gemini-CLI | Claim only | robots.txt:No policy | ||
| opencode | opencode (open source) | Claim only | robots.txt:No policy | |
| Trae | ByteDance | Claim only | robots.txt:No policy |
Collect pages into datasets that may be reused, licensed or shared.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| AgentTimes | The Agent Times | Claim only | robots.txt:No policy | |
| aiHitBot | aiHit | Claim only | robots.txt:Honours | |
| ApifyBot | Unidentified | Claim only | robots.txt:No policy | |
| ApifyWebsiteContentCrawler | Unidentified | Claim only | robots.txt:No policy | |
| Awario | Awario | Claim only | robots.txt:No policy | |
| bedrockbot | Amazon | Claim only | robots.txt:Honours | |
| Bravebot | Brave | Claim only | robots.txt:Honours | |
| BrightbotBrightbot 1.0 | Unidentified | Claim only | robots.txt:No policy | |
| CCBot | Common Crawl Foundation | Claim only | robots.txt:Honours | |
| ChatGLM-Spider | Zhipu AI | Claim only | robots.txt:No policy | |
| cohere-training-data-crawler | Cohere | Claim only | robots.txt:No policy | |
| CragCrawler | Unidentified | Claim only | robots.txt:No policy | |
| Crawlspace | Crawlspace | Claim only | robots.txt:Honours | |
| Datenbank Crawler | Datenbank | Claim only | robots.txt:No policy | |
| Diffbot | Diffbot | Claim only | robots.txt:No policy | |
| Echobot Bot | Echobox | Claim only | robots.txt:No policy | |
| ExaBot | Exa | Claim only | robots.txt:No policy | |
| FirecrawlAgent | Firecrawl | Claim only | robots.txt:Honours | |
| HenkBot | Unidentified | Claim only | robots.txt:No policy | |
| imageSpider | Unidentified | Claim only | robots.txt:No policy | |
| Kangaroo Bot | Kangaroo LLM | Claim only | robots.txt:No policy | |
| laion-huggingface-processor | LAION | Claim only | robots.txt:No policy | |
| LAIONDownloader | Large-scale Artificial Intelligence Open Network | Claim only | robots.txt:Ignores | |
| LCC | Unidentified | Claim only | robots.txt:No policy | |
| Lightpanda | Unidentified | Claim only | robots.txt:No policy | |
| Mozilla-Tabstack | Mozilla | Claim only | robots.txt:Honours | |
| MyCentralAIScraperBot | Unidentified | Claim only | robots.txt:No policy | |
| netEstate Imprint Crawler | netEstate | Claim only | robots.txt:No policy | |
| omgiliomgilibot | Webz.io | Claim only | robots.txt:Honours | |
| PanguBot | the Chinese company Huawei | Claim only | robots.txt:No policy | |
| QueritBotQuerit-SearchBot | Querit | Claim only | robots.txt:No policy | |
| SemrushBot-OCOB | Semrush | Claim only | robots.txt:Honours | |
| ShapBot | Parallel | Claim only | robots.txt:Honours | |
| Spider | Unidentified | Claim only | robots.txt:No policy | |
| TavilyBot | Tavily | Claim only | robots.txt:No policy | |
| TerraCottaTerra Cotta | Ceramic AI | Claim only | robots.txt:Honours | |
| VelenPublicWebCrawler | Velen Crawler | Claim only | robots.txt:Honours | |
| WARDBot | WEBSPARK | Claim only | robots.txt:No policy | |
| Webzio-Extendedwebzio-extended | Unidentified | Claim only | robots.txt:No policy | |
| YandexAdditionalYandexAdditionalBot | Yandex | Reverse DNS | robots.txt:Honours |
Bots in AI workflows that fit none of the above.
| Crawler | Operator | Verification | robots.txt | Details |
|---|---|---|---|---|
| BuddyBot | BuddyBotLearning | Claim only | robots.txt:No policy | |
| EchoboxBot | Echobox | Claim only | robots.txt:No policy | |
| facebookexternalhit | Meta/Facebook | Operator ASN | robots.txt:Ignores | |
| Google-Firebase | Claim only | robots.txt:No policy | ||
| OpenAI | OpenAI | IP ranges | robots.txt:Honours | |
| Panscientpanscient.com | Panscient | Claim only | robots.txt:Honours | |
| Poseidon Research Crawler | Poseidon Research | Claim only | robots.txt:No policy | |
| QuillBotquillbot.com | Quillbot | Claim only | robots.txt:No policy | |
| SBIntuitionsBot | SB Intuitions | Claim only | robots.txt:Honours | |
| Thinkbot | Thinkbot | Claim only | robots.txt:Ignores | |
| wpbot | QuantumCloud | Claim only | robots.txt:No policy | |
| YaK | Meltwater | Claim only | robots.txt:No policy |
robots.txt
One rule per token, aliases included. 9 of these ignore the file, so pair it with edge rules.
User-agent: AddSearchBot
User-agent: AgentTimes
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
User-agent: aiHitBot
User-agent: AIWebIndex
User-agent: AIWebIndex-Agent
User-agent: amazon-kendra
User-agent: amazon-QBusiness
User-agent: Amazonbot
User-agent: AmazonBuyForMe
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Andibot
User-agent: Anomura
User-agent: anthropic-ai
User-agent: ApifyBot
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Aranet-SearchBot
User-agent: atlassian-bot
User-agent: Awario
User-agent: AzureAI-SearchBot
User-agent: bedrockbot
User-agent: bigsur.ai
User-agent: Bingbot
User-agent: Bravebot
User-agent: Brightbot
User-agent: Brightbot 1.0
User-agent: BuddyBot
User-agent: Bytespider
User-agent: CCBot
User-agent: Channel3Bot
User-agent: ChatGLM-Spider
User-agent: ChatGPT Agent
User-agent: ChatGPT-User
User-agent: Claude-Code
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cloudflare-AutoRAG
User-agent: CloudVertexBot
User-agent: Code
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: CragCrawler
User-agent: Crawl4AI
User-agent: Crawlspace
User-agent: Cursor
User-agent: Datenbank Crawler
User-agent: DeepSeekBot
User-agent: Devin
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: DuckDuckBot
User-agent: Echobot Bot
User-agent: EchoboxBot
User-agent: ExaBot
User-agent: ExaSearchBot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: FriendlyCrawler
User-agent: GeistHaus-PageFetcher
User-agent: Gemini-Deep-Research
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Gemini-CLI
User-agent: Google-NotebookLM
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Googlebot
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: HenkBot
User-agent: iAskBot
User-agent: iaskspider
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: imageSpider
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: kagi-fetcher
User-agent: Kangaroo Bot
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: KunatoCrawler
User-agent: laion-huggingface-processor
User-agent: LAIONDownloader
User-agent: LCC
User-agent: Lightpanda
User-agent: LinerBot
User-agent: Linguee Bot
User-agent: LinkupBot
User-agent: Manus-User
User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: meta-webindexer
User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
User-agent: Mozilla-Tabstack
User-agent: MyCentralAIScraperBot
User-agent: NagetBot
User-agent: netEstate Imprint Crawler
User-agent: newsai
User-agent: NotebookLM
User-agent: NovaAct
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: OpenAI
User-agent: opencode
User-agent: Operator
User-agent: PanguBot
User-agent: Panscient
User-agent: panscient.com
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: Poggio-Citations
User-agent: Poseidon Research Crawler
User-agent: QualifiedBot
User-agent: Querit-SearchBot
User-agent: QueritBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: Reflectionbot
User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Shap-User
User-agent: ShapBot
User-agent: Sidetrade indexer bot
User-agent: Spider
User-agent: TavilyBot
User-agent: Terra Cotta
User-agent: TerraCotta
User-agent: Thinkbot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: TongyiBot
User-agent: Trae
User-agent: TwinAgent
User-agent: UseAI
User-agent: VelenPublicWebCrawler
User-agent: WARDBot
User-agent: webzio-extended
User-agent: Webzio-Extended
User-agent: wpbot
User-agent: WRTNBot
User-agent: YaK
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: YiyanBot
User-agent: YouBot
User-agent: ZanistaBot
Disallow: /Or start from a preset
Log check
Paste a hundred lines. Every crawler claim in them is checked against what its operator publishes.
FAQ
9 of the 153 tokens listed here are documented as ignoring robots.txt: Bytespider, DeepSeekBot, facebookexternalhit, ISSCyberRiskCrawler, LAIONDownloader, Linguee Bot, Meta-ExternalFetcher, Perplexity-User, Thinkbot. A further 90 have no published policy either way, which is not the same as a promise to obey it. For those, a robots.txt rule is a request and an edge rule is enforcement.
By checking the source IP, never the user agent. Three things are checkable: a published IP range file, a forward-confirmed reverse DNS convention, or an operator ASN. Only 34 of the 153 tokens here publish any of them; for the rest there is nothing to check against.
No. The user agent is a header the client chooses, so anything can send it. Scrapers copy the strings of well-known crawlers precisely because site owners allow those through. Treat the user agent as a claim and the IP as the evidence.
No. Blocking stops future crawls; it does not remove anything already collected, and it has no effect on what a model already learned. It also does not change how the site ranks in ordinary search.
A training crawler collects pages that end up in a model's training corpus. An AI search crawler builds the index an answer engine cites from. Blocking the first is the opt-out most publishers mean; blocking the second removes you from the answers.
IPScanner (2026). AI crawler directory. https://ipscanner.io/bot. CC BY 4.0.