AI search crawlers
OAI-SearchBot is a bot operated by OpenAI that indexes pages so an AI answer engine can cite them. Crawls sites to surface as results in SearchGPT. It honours robots.txt, and a request claiming to be OAI-SearchBot can be checked against a published IP range file.
Request says
"OAI-SearchBot/1.0"
Checked against
OpenAI's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot |
|---|---|
| Operator | OpenAI |
| Purpose | Search result generation. |
| Respects robots.txt | Honourssource |
| Verification method | IP rangesPublished IP range file OpenAI publishes a separate range file for the search crawler. Blocking it removes the site from ChatGPT search results, which is a different decision from blocking training. |
| Crawl pattern | Not published |
| Robots.txt tokens | OAI-SearchBot |
| Operator documentation | https://platform.openai.com/docs/bots |
Crawls sites to surface as results in SearchGPT.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: OAI-SearchBot
Disallow: /User-agent: OAI-SearchBot
Allow: /User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking OAI-SearchBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
OAI-SearchBot is a bot operated by OpenAI that indexes pages so an AI answer engine can cite them. Crawls sites to surface as results in SearchGPT. It honours robots.txt, and a request claiming to be OAI-SearchBot can be checked against a published IP range file.
Yes. The operator documents that OAI-SearchBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
OpenAI publishes a separate range file for the search crawler. Blocking it removes the site from ChatGPT search results, which is a different decision from blocking training. The reference is published at https://openai.com/searchbot.json.
Add the OAI-SearchBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking OAI-SearchBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.