Which AI crawlers ignore robots.txt?

9 of the 153 tokens listed here are documented as ignoring robots.txt: Bytespider, DeepSeekBot, facebookexternalhit, ISSCyberRiskCrawler, LAIONDownloader, Linguee Bot, Meta-ExternalFetcher, Perplexity-User, Thinkbot. A further 90 have no published policy either way, which is not the same as a promise to obey it. For those, a robots.txt rule is a request and an edge rule is enforcement.

How do you verify that an AI crawler is genuine?

By checking the source IP, never the user agent. Three things are checkable: a published IP range file, a forward-confirmed reverse DNS convention, or an operator ASN. Only 34 of the 153 tokens here publish any of them; for the rest there is nothing to check against.

Is the user-agent string enough to identify a crawler?

No. The user agent is a header the client chooses, so anything can send it. Scrapers copy the strings of well-known crawlers precisely because site owners allow those through. Treat the user agent as a claim and the IP as the evidence.

Does blocking a training crawler remove my content from the model?

No. Blocking stops future crawls; it does not remove anything already collected, and it has no effect on what a model already learned. It also does not change how the site ranks in ordinary search.

What is the difference between a training crawler and an AI search crawler?

A training crawler collects pages that end up in a model's training corpus. An AI search crawler builds the index an answer engine cites from. Blocking the first is the opt-out most publishers mean; blocking the second removes you from the answers.

IPScanner

AI crawler directory

Every AI crawler, and how to tell if it is real

119 of the 153 AI crawler tokens in this directory publish no way to prove a request is really theirs: no IP range file, no reverse-DNS convention, no operator ASN. For those 78%, the user agent is a claim and nothing more, and anyone can type it. The remaining 34 can be checked, and each entry below says exactly how: who runs the crawler, the string as it appears in your logs, whether the operator has committed to robots.txt, and what evidence exists either way.

78%

publish no way to prove a request is really theirs

153

crawler tokens tracked in total

documented as ignoring robots.txt

with no published robots.txt policy

Base facts come from the ai.robots.txt community list and from operator documentation. User-agent strings, IP range files and reverse-DNS conventions were checked against the operators' own published sources. Where nothing is published, the entry says so instead of guessing.

Last updated 2026年8月24日

robots.txt

Block every AI crawler at once

One rule per token, covering every entry in this directory including the legacy aliases. Paste it at the root of your robots.txt. Bear in mind that 9 of these are documented as ignoring the file entirely, so pair it with edge rules if the traffic actually costs you money.

Block all 153 crawlers

User-agent: AddSearchBot
User-agent: AgentTimes
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
User-agent: aiHitBot
User-agent: AIWebIndex
User-agent: AIWebIndex-Agent
User-agent: amazon-kendra
User-agent: amazon-QBusiness
User-agent: Amazonbot
User-agent: AmazonBuyForMe
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Andibot
User-agent: Anomura
User-agent: anthropic-ai
User-agent: ApifyBot
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Aranet-SearchBot
User-agent: atlassian-bot
User-agent: Awario
User-agent: AzureAI-SearchBot
User-agent: bedrockbot
User-agent: bigsur.ai
User-agent: Bingbot
User-agent: Bravebot
User-agent: Brightbot
User-agent: Brightbot 1.0
User-agent: BuddyBot
User-agent: Bytespider
User-agent: CCBot
User-agent: Channel3Bot
User-agent: ChatGLM-Spider
User-agent: ChatGPT Agent
User-agent: ChatGPT-User
User-agent: Claude-Code
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cloudflare-AutoRAG
User-agent: CloudVertexBot
User-agent: Code
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: CragCrawler
User-agent: Crawl4AI
User-agent: Crawlspace
User-agent: Cursor
User-agent: Datenbank Crawler
User-agent: DeepSeekBot
User-agent: Devin
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: DuckDuckBot
User-agent: Echobot Bot
User-agent: EchoboxBot
User-agent: ExaBot
User-agent: ExaSearchBot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: FriendlyCrawler
User-agent: GeistHaus-PageFetcher
User-agent: Gemini-Deep-Research
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Gemini-CLI
User-agent: Google-NotebookLM
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Googlebot
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: HenkBot
User-agent: iAskBot
User-agent: iaskspider
User-agent: iaskspider/2.0
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: imageSpider
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: kagi-fetcher
User-agent: Kangaroo Bot
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: KunatoCrawler
User-agent: laion-huggingface-processor
User-agent: LAIONDownloader
User-agent: LCC
User-agent: Lightpanda
User-agent: LinerBot
User-agent: Linguee Bot
User-agent: LinkupBot
User-agent: Manus-User
User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: meta-webindexer
User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
User-agent: Mozilla-Tabstack
User-agent: MyCentralAIScraperBot
User-agent: NagetBot
User-agent: netEstate Imprint Crawler
User-agent: newsai
User-agent: NotebookLM
User-agent: NovaAct
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: OpenAI
User-agent: opencode
User-agent: Operator
User-agent: PanguBot
User-agent: Panscient
User-agent: panscient.com
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: Poggio-Citations
User-agent: Poseidon Research Crawler
User-agent: QualifiedBot
User-agent: Querit-SearchBot
User-agent: QueritBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: Reflectionbot
User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Shap-User
User-agent: ShapBot
User-agent: Sidetrade indexer bot
User-agent: Spider
User-agent: TavilyBot
User-agent: Terra Cotta
User-agent: TerraCotta
User-agent: Thinkbot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: TongyiBot
User-agent: Trae
User-agent: TwinAgent
User-agent: UseAI
User-agent: VelenPublicWebCrawler
User-agent: WARDBot
User-agent: webzio-extended
User-agent: Webzio-Extended
User-agent: wpbot
User-agent: WRTNBot
User-agent: YaK
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: YiyanBot
User-agent: YouBot
User-agent: ZanistaBot
Disallow: /

Block training crawlers only

User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: FacebookBot
User-agent: Factset_spyderbot
User-agent: FriendlyCrawler
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ICC-Crawler
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Linguee Bot
User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: Scrapy
User-agent: Sidetrade indexer bot
User-agent: TikTokSpider
Disallow: /

Keeps AI search crawlers, so you can still be cited in answers.

Block dataset builders only

User-agent: AgentTimes
User-agent: aiHitBot
User-agent: ApifyBot
User-agent: ApifyWebsiteContentCrawler
User-agent: Awario
User-agent: bedrockbot
User-agent: Bravebot
User-agent: Brightbot
User-agent: Brightbot 1.0
User-agent: CCBot
User-agent: ChatGLM-Spider
User-agent: cohere-training-data-crawler
User-agent: CragCrawler
User-agent: Crawlspace
User-agent: Datenbank Crawler
User-agent: Diffbot
User-agent: Echobot Bot
User-agent: ExaBot
User-agent: FirecrawlAgent
User-agent: HenkBot
User-agent: imageSpider
User-agent: Kangaroo Bot
User-agent: laion-huggingface-processor
User-agent: LAIONDownloader
User-agent: LCC
User-agent: Lightpanda
User-agent: Mozilla-Tabstack
User-agent: MyCentralAIScraperBot
User-agent: netEstate Imprint Crawler
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: Querit-SearchBot
User-agent: QueritBot
User-agent: SemrushBot-OCOB
User-agent: ShapBot
User-agent: Spider
User-agent: TavilyBot
User-agent: Terra Cotta
User-agent: TerraCotta
User-agent: VelenPublicWebCrawler
User-agent: WARDBot
User-agent: webzio-extended
User-agent: Webzio-Extended
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /

Keeps your pages out of collected datasets while leaving the model vendors alone.

Training crawlers21

Collect pages that end up in a model's training corpus. Blocking these is the opt-out most publishers mean when they say they blocked AI.

Training crawlers: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
AI2BotAi2Bot-Dolma	Ai2	Yes, documented	None published, user agent only
anthropic-ai	Anthropic	Not documented	Published IP range file
Applebot-Extended	Apple	Yes, documented	Not applicable, robots.txt token only
Bytespider	ByteDance	No	None published, user agent only
ClaudeBot	Anthropic	Yes, documented	Published IP range file
cohere-ai	Cohere	Not documented	None published, user agent only
Cotoyogi	ROIS	Yes, documented	None published, user agent only
DeepSeekBot	DeepSeek	No	None published, user agent only
FacebookBot	Meta/Facebook	Yes, documented	Operator ASN check
Factset_spyderbot	Factset	Not documented	None published, user agent only
FriendlyCrawler	Unknown	Yes, documented	None published, user agent only
Google-Extended	Google	Yes, documented	Not applicable, robots.txt token only
GPTBot	OpenAI	Yes, documented	Published IP range file
ICC-Crawler	NICT	Yes, documented	None published, user agent only
img2dataset	img2dataset	Not documented	None published, user agent only
ISSCyberRiskCrawler	ISS-Corporate	No	None published, user agent only
Linguee Bot	Linguee	No	None published, user agent only
Meta-ExternalAgentmeta-externalagent	Meta	Yes, documented	Operator ASN check
Scrapy	Zyte	Not documented	None published, user agent only
Sidetrade indexer bot	Sidetrade	Not documented	None published, user agent only
TikTokSpider	ByteDance	Not documented	None published, user agent only

AI search crawlers29

Build the index an AI answer engine cites from. Blocking one removes you from its answers, which is usually not what you want.

AI search crawlers: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
AddSearchBot	Unidentified	Not documented	None published, user agent only
AIWebIndexAIWebIndex-Agent	Lyrenth (Aleksma Ai, Inc.)	Yes, documented	Published IP ranges or reverse DNS
amazon-kendra	Amazon	Yes, documented	None published, user agent only
Amazonbot	Amazon	Yes, documented	Reverse DNS, forward confirmed
Amzn-SearchBot	Amazon	Not documented	Reverse DNS, forward confirmed
Andibot	Andi	Not documented	None published, user agent only
Anomura	Direqt	Yes, documented	None published, user agent only
Applebot	Apple	Yes, documented	Published IP ranges or reverse DNS
atlassian-bot	Atlassian	Yes, documented	None published, user agent only
AzureAI-SearchBot	Microsoft	Not documented	None published, user agent only
Bingbot	Microsoft	Yes, documented	Published IP ranges or reverse DNS
Channel3Bot	Unidentified	Not documented	None published, user agent only
Claude-SearchBot	Anthropic	Yes, documented	Published IP range file
Cloudflare-AutoRAG	Cloudflare	Yes, documented	None published, user agent only
DuckDuckBot	DuckDuckGo	Yes, documented	Published IP range file
ExaSearchBot	Exa	Not documented	None published, user agent only
Googlebot	Google	Yes, documented	Published IP ranges or reverse DNS
GoogleOther	Google	Yes, documented	Published IP ranges or reverse DNS
GoogleOther-Image	Google	Yes, documented	Published IP ranges or reverse DNS
GoogleOther-Video	Google	Yes, documented	Published IP ranges or reverse DNS
ImagesiftBot	ImageSift	Yes, documented	None published, user agent only
LinkupBot	Linkup	Not documented	None published, user agent only
meta-webindexer	Meta	Not documented	Operator ASN check
OAI-SearchBot	OpenAI	Yes, documented	Published IP range file
PerplexityBot	Perplexity	Yes, documented	Published IP range file
PetalBot	Huawei	Yes, documented	None published, user agent only
Timpibot	Timpi	Not documented	None published, user agent only
YouBot	You	Yes, documented	None published, user agent only
ZanistaBot	Unidentified	Not documented	None published, user agent only

Assistant fetchers26

Fetch a single page because a person asked an assistant about it. Traffic is spiky, and some operators document that these user-requested fetches skip robots.txt.

Assistant fetchers: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
AI2Bot-DeepResearchEval	Ai2, a non-profit AI research institute	Not documented	None published, user agent only
amazon-QBusiness	Amazon Web Services	Not documented	None published, user agent only
Amzn-User	Amazon	Not documented	Reverse DNS, forward confirmed
bigsur.ai	Unidentified	Not documented	None published, user agent only
ChatGPT-User	OpenAI	Yes, documented	Published IP range file
Claude-User	Anthropic	Yes, documented	Published IP range file
DuckAssistBot	Unidentified	Yes, documented	Published IP range file
GeistHaus-PageFetcher	Unidentified	Not documented	None published, user agent only
Gemini-Deep-Research	Google	Not documented	None published, user agent only
Google-NotebookLMNotebookLM	Google	Not documented	Published IP range file
GoogleAgent-URLContext	Google	Not documented	None published, user agent only
kagi-fetcher	Unidentified	Not documented	None published, user agent only
Kimi-User	Moonshot AI	Not documented	None published, user agent only
KlaviyoAIBot	Klaviyo	Yes, documented	None published, user agent only
LinerBot	Unidentified	Not documented	None published, user agent only
Meta-ExternalFetchermeta-externalfetcher	Meta	No	Operator ASN check
MistralAI-UserMistralAI-User/1.0	Mistral	Not documented	None published, user agent only
Perplexity-User	Perplexity	No	Published IP range file
PhindBot	phind	Not documented	None published, user agent only
Poggio-Citations	Unidentified	Not documented	None published, user agent only
QualifiedBot	Qualified	Not documented	None published, user agent only
SemrushBot-SWA	Semrush	Yes, documented	None published, user agent only
Shap-User	Parallel	Not documented	None published, user agent only
TongyiBot	Alibaba	Not documented	None published, user agent only
UseAI	Unidentified	Not documented	None published, user agent only
YiyanBot	Baidu that fetches web content for the yiyan	Not documented	None published, user agent only

Autonomous agents18

Drive a browser through multi-step tasks. They look like a human session until you check the headers or the source IP.

Autonomous agents: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
AmazonBuyForMe	Amazon	Not documented	None published, user agent only
Aranet-SearchBot	Unidentified	Not documented	None published, user agent only
ChatGPT Agent	OpenAI	Yes, documented	Published IP range file
Claude-Web	Anthropic	Not documented	Published IP range file
Crawl4AI	Crawl4AI (open source)	Not documented	None published, user agent only
Google-Agent	Google	Yes, documented	None published, user agent only
Google-CloudVertexBotCloudVertexBot	Google	Yes, documented	Published IP range file
GoogleAgent-Mariner	Google	Not documented	None published, user agent only
iAskBotiaskspider, iaskspider/2.0	iAsk	Not documented	None published, user agent only
KunatoCrawler	Unidentified	Not documented	None published, user agent only
Manus-User	Butterfly Effect, a company based in China	Not documented	None published, user agent only
NagetBot	Unidentified	Not documented	None published, user agent only
newsai	Unidentified	Not documented	None published, user agent only
NovaAct	Amazon	Not documented	None published, user agent only
Operator	OpenAI	Not documented	Published IP range file
Reflectionbot	Reflection	Not documented	None published, user agent only
TwinAgent	Unidentified	Not documented	None published, user agent only
WRTNBot	Wrtn Technologies	Not documented	None published, user agent only

Coding agents7

Pull docs and examples while writing code. The source IP is usually a developer laptop or a CI runner, not the vendor.

Coding agents: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
Claude-Code	Anthropic	Not documented	Published IP range file
Code	GitHub Copilot	Not documented	None published, user agent only
Cursor	Cursor	Not documented	None published, user agent only
Devin	Devin AI	Yes, documented	None published, user agent only
Google-Gemini-CLI	Google	Not documented	None published, user agent only
opencode	opencode (open source)	Not documented	None published, user agent only
Trae	ByteDance	Not documented	None published, user agent only

Dataset builders40

Collect pages into datasets that may be reused, licensed or redistributed. Blocking limits future collection rather than one model reading the page.

Dataset builders: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
AgentTimes	The Agent Times	Not documented	None published, user agent only
aiHitBot	aiHit	Yes, documented	None published, user agent only
ApifyBot	Unidentified	Not documented	None published, user agent only
ApifyWebsiteContentCrawler	Unidentified	Not documented	None published, user agent only
Awario	Awario	Not documented	None published, user agent only
bedrockbot	Amazon	Yes, documented	None published, user agent only
Bravebot	Brave	Yes, documented	None published, user agent only
BrightbotBrightbot 1.0	Unidentified	Not documented	None published, user agent only
CCBot	Common Crawl Foundation	Yes, documented	None published, user agent only
ChatGLM-Spider	Zhipu AI	Not documented	None published, user agent only
cohere-training-data-crawler	Cohere	Not documented	None published, user agent only
CragCrawler	Unidentified	Not documented	None published, user agent only
Crawlspace	Crawlspace	Yes, documented	None published, user agent only
Datenbank Crawler	Datenbank	Not documented	None published, user agent only
Diffbot	Diffbot	Not documented	None published, user agent only
Echobot Bot	Echobox	Not documented	None published, user agent only
ExaBot	Exa	Not documented	None published, user agent only
FirecrawlAgent	Firecrawl	Yes, documented	None published, user agent only
HenkBot	Unidentified	Not documented	None published, user agent only
imageSpider	Unidentified	Not documented	None published, user agent only
Kangaroo Bot	Kangaroo LLM	Not documented	None published, user agent only
laion-huggingface-processor	LAION	Not documented	None published, user agent only
LAIONDownloader	Large-scale Artificial Intelligence Open Network	No	None published, user agent only
LCC	Unidentified	Not documented	None published, user agent only
Lightpanda	Unidentified	Not documented	None published, user agent only
Mozilla-Tabstack	Mozilla	Yes, documented	None published, user agent only
MyCentralAIScraperBot	Unidentified	Not documented	None published, user agent only
netEstate Imprint Crawler	netEstate	Not documented	None published, user agent only
omgiliomgilibot	Webz.io	Yes, documented	None published, user agent only
PanguBot	the Chinese company Huawei	Not documented	None published, user agent only
QueritBotQuerit-SearchBot	Querit	Not documented	None published, user agent only
SemrushBot-OCOB	Semrush	Yes, documented	None published, user agent only
ShapBot	Parallel	Yes, documented	None published, user agent only
Spider	Unidentified	Not documented	None published, user agent only
TavilyBot	Tavily	Not documented	None published, user agent only
TerraCottaTerra Cotta	Ceramic AI	Yes, documented	None published, user agent only
VelenPublicWebCrawler	Velen Crawler	Yes, documented	None published, user agent only
WARDBot	WEBSPARK	Not documented	None published, user agent only
Webzio-Extendedwebzio-extended	Unidentified	Not documented	None published, user agent only
YandexAdditionalYandexAdditionalBot	Yandex	Yes, documented	Reverse DNS, forward confirmed

Other automated crawlers12

Bots that touch AI workflows without fitting the categories above.

Other automated crawlers: user-agent token, operator, robots.txt policy and how each one can be verified
Crawler	Operator	robots.txt	Verification
BuddyBot	BuddyBotLearning	Not documented	None published, user agent only
EchoboxBot	Echobox	Not documented	None published, user agent only
facebookexternalhit	Meta/Facebook	No	Operator ASN check
Google-Firebase	Google	Not documented	None published, user agent only
OpenAI	OpenAI	Yes, documented	Published IP range file
Panscientpanscient.com	Panscient	Yes, documented	None published, user agent only
Poseidon Research Crawler	Poseidon Research	Not documented	None published, user agent only
QuillBotquillbot.com	Quillbot	Not documented	None published, user agent only
SBIntuitionsBot	SB Intuitions	Yes, documented	None published, user agent only
Thinkbot	Thinkbot	No	None published, user agent only
wpbot	QuantumCloud	Not documented	None published, user agent only
YaK	Meltwater	Not documented	None published, user agent only

Check your own access log

Paste a hundred lines and every crawler claim in them is checked against what that operator publishes, the same way a single address is checked on a crawler's own page. Most logs come back with more unverifiable claims than false ones, which is its own finding: it is not that the traffic is proven fake, it is that nobody published a way to prove it real.

Checking a log needs JavaScript, because the file is read in your browser and only the address and user agent from each line are ever sent. Without it, look a token up in the directory above and check a single address on that crawler's own page.

Questions this directory answers

Which AI crawlers ignore robots.txt?: 9 of the 153 tokens listed here are documented as ignoring robots.txt: Bytespider, DeepSeekBot, facebookexternalhit, ISSCyberRiskCrawler, LAIONDownloader, Linguee Bot, Meta-ExternalFetcher, Perplexity-User, Thinkbot. A further 90 have no published policy either way, which is not the same as a promise to obey it. For those, a robots.txt rule is a request and an edge rule is enforcement.
How do you verify that an AI crawler is genuine?: By checking the source IP, never the user agent. Three things are checkable: a published IP range file, a forward-confirmed reverse DNS convention, or an operator ASN. Only 34 of the 153 tokens here publish any of them; for the rest there is nothing to check against.
Is the user-agent string enough to identify a crawler?: No. The user agent is a header the client chooses, so anything can send it. Scrapers copy the strings of well-known crawlers precisely because site owners allow those through. Treat the user agent as a claim and the IP as the evidence.
Does blocking a training crawler remove my content from the model?: No. Blocking stops future crawls; it does not remove anything already collected, and it has no effect on what a model already learned. It also does not change how the site ranks in ordinary search.
What is the difference between a training crawler and an AI search crawler?: A training crawler collects pages that end up in a model's training corpus. An AI search crawler builds the index an answer engine cites from. Blocking the first is the opt-out most publishers mean; blocking the second removes you from the answers.

Open data

Cite this dataset

Every row in this directory, 153 crawler tokens with their operator, user-agent string, robots.txt stance and verification method, as one file. Published under CC BY 4.0: use it anywhere, including commercially, as long as the credit line stays with it.

agents.jsonVersioned object with per agent records agents.csvOne row per agent, for a spreadsheet

Dataset updated 2026年8月24日. The files are rebuilt from the same source as the table above, so the two never disagree.

IPScanner (2026). AI crawler directory. https://ipscanner.io/bot. CC BY 4.0.

Agentscan

Stop maintaining range files by hand

Every operator publishes its ranges in a different place, in a different format, and changes them without notice. Agentscan does the reverse-DNS and range matching for you and returns a verdict per request, so your edge can allow the real search crawlers and challenge everything wearing their names.

How Agentscan works Verify Googlebot API reference Datacenter IP detection IP reputation check Pricing