AI search crawlers
YouBot is a bot operated by You that indexes pages so an AI answer engine can cite them. Retrieves data used for You.com web search engine and LLMs. It is documented as honouring robots.txt, but no IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine.
Request says
"YouBot"
Checked against
Nothing published
Live check
No operator check exists for YouBot. This still shows what the address really is.
Reference
| User agent | YouBotToken only: match on the substring. |
|---|---|
| Operator | You |
| Purpose | Scrapes data for search engine and LLMs. |
| Respects robots.txt | Honourssource |
| Verification method | Claim onlyNone published, user agent only You.com documents the crawler and honours robots.txt but publishes no address list. |
| Crawl pattern | Not published |
| Robots.txt tokens | YouBot |
| Operator documentation | https://about.you.com/youbot/ |
Retrieves data used for You.com web search engine and LLMs.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: YouBot
Disallow: /User-agent: YouBot
Allow: /User-agent: YouBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking YouBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
YouBot is a bot operated by You that indexes pages so an AI answer engine can cite them. Retrieves data used for You.com web search engine and LLMs. It is documented as honouring robots.txt, but no IP range list or reverse-DNS convention is published, so a request carrying this user agent cannot be proven genuine.
Yes. The operator documents that YouBot reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
You cannot, at least not from the request itself. You.com documents the crawler and honours robots.txt but publishes no address list. Compare the source IP against hosting and datacenter ranges instead, and rate limit anything that behaves like a crawler.
Add the YouBot token to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking YouBot takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.