Other automated crawlers
OpenAI is a bot that crawls the web automatically. The purpose of this bot is unclear at this time but it is a member of OpenAI's suite of crawlers. It honours robots.txt, and a request claiming to be OpenAI can be checked against a published IP range file.
Request says
"OpenAI"
Checked against
OpenAI's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | OpenAIToken only: match on the substring. |
|---|---|
| Operator | OpenAI |
| Purpose | Other automated crawlers |
| Respects robots.txt | Honours |
| Verification method | IP rangesPublished IP range file A bare "OpenAI" token is not one of the three documented crawlers. Treat it as unverified unless the source IP appears in one of the published OpenAI range files. |
| Crawl pattern | Not published |
| Robots.txt tokens | OpenAI |
| Operator documentation | https://platform.openai.com/docs/bots |
The purpose of this bot is unclear at this time but it is a member of OpenAI's suite of crawlers.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: OpenAI
Disallow: /User-agent: OpenAI
Allow: /User-agent: OpenAI
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking OpenAI stops this specific token. Check your logs afterwards, because operators that ignore robots.txt often come back under a different name.
Build a file for every crawlerFAQ
OpenAI is a bot that crawls the web automatically. The purpose of this bot is unclear at this time but it is a member of OpenAI's suite of crawlers. It honours robots.txt, and a request claiming to be OpenAI can be checked against a published IP range file.
Yes. The operator documents that OpenAI reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
A bare "OpenAI" token is not one of the three documented crawlers. Treat it as unverified unless the source IP appears in one of the published OpenAI range files. The reference is published at https://platform.openai.com/docs/bots.
Add the OpenAI token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking OpenAI stops this specific token. Check your logs afterwards, because operators that ignore robots.txt often come back under a different name.