Training crawlers
GPTBot is a bot operated by OpenAI that collects pages used to train AI models. Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies. It honours robots.txt, and a request claiming to be GPTBot can be checked against a published IP range file.
Request says
"GPTBot/1.2"
Checked against
OpenAI's published IP ranges
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot |
|---|---|
| Operator | OpenAI |
| Purpose | Scrapes data to train OpenAI's products. |
| Respects robots.txt | Honours |
| Verification method | IP rangesPublished IP range file OpenAI publishes the GPTBot egress ranges as a JSON file. Match the request IP against that file; there is no reverse-DNS convention. |
| Crawl pattern | Not published |
| Robots.txt tokens | GPTBot |
| Operator documentation | https://platform.openai.com/docs/bots |
Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: GPTBot
Disallow: /User-agent: GPTBot
Allow: /User-agent: GPTBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking GPTBot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.
Build a file for every crawlerFAQ
GPTBot is a bot operated by OpenAI that collects pages used to train AI models. Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies. It honours robots.txt, and a request claiming to be GPTBot can be checked against a published IP range file.
Yes. The operator documents that GPTBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
OpenAI publishes the GPTBot egress ranges as a JSON file. Match the request IP against that file; there is no reverse-DNS convention. The reference is published at https://openai.com/gptbot.json.
Add the GPTBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking GPTBot keeps your pages out of the next training run. It does not remove anything already collected, and it has no effect on how the site ranks in ordinary search.