AI search crawlers
AIWebIndex is a bot operated by Lyrenth (Aleksma Ai, Inc.) that indexes pages so an AI answer engine can cite them. AIWebIndex is a web crawler operated by Lyrenth that builds an AI-readable index of web content and serves pages to AI agents with attribution. It honours robots.txt, and a request claiming to be AIWebIndex can be checked against a published IP range file or a forward-confirmed reverse DNS lookup.
Request says
"AIWebIndex/2.0"
Checked against
Lyrenth (Aleksma Ai, Inc.)'s IP ranges and reverse DNS
Live check
Paste the address from the log line. It is checked against what the operator publishes.
Reference
| User agent | AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index) |
|---|---|
| Operator | Lyrenth (Aleksma Ai, Inc.) |
| Purpose | Builds an AI-readable web index and serves individual pages back to AI agents as structured text with attribution and a link to the source. |
| Respects robots.txt | Honourssource |
| Verification method | IP ranges + DNSPublished IP ranges or reverse DNS Lyrenth publishes its crawl IPs as JSON (gptbot.json format). Reverse DNS resolves under lyrenth.com and forward-resolves back to the same IP. A Web Bot Auth (RFC 9421) key directory is also published at api.lyrenth.com/.well-known/http-message-signatures-directory. All 22 published addresses forward-confirmed on 2026-08-24. |
| Crawl pattern | At most one request per domain every 2 seconds, and slower where robots.txt sets a longer Crawl-delay. |
| Robots.txt tokens | AIWebIndexAIWebIndex-Agent |
| Operator documentation | https://lyrenth.com/crawler-policy |
AIWebIndex is a web crawler operated by Lyrenth that builds an AI-readable index of web content and serves pages to AI agents with attribution. Lyrenth states it does not train foundation models on crawled content. Its crawler policy notes that owners who verify a domain grant Lyrenth the right to license that domain's canonical content to AI labs; this applies to verified domains only, not to regular crawling.
robots.txt
Paste one at the root of your domain. Rules are per token.
User-agent: AIWebIndex
User-agent: AIWebIndex-Agent
Disallow: /User-agent: AIWebIndex
User-agent: AIWebIndex-Agent
Allow: /User-agent: AIWebIndex
User-agent: AIWebIndex-Agent
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /searchEdit the Disallow paths to match your own URLs.
Blocking AIWebIndex takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.
Build a file for every crawlerFAQ
AIWebIndex is a bot operated by Lyrenth (Aleksma Ai, Inc.) that indexes pages so an AI answer engine can cite them. AIWebIndex is a web crawler operated by Lyrenth that builds an AI-readable index of web content and serves pages to AI agents with attribution. It honours robots.txt, and a request claiming to be AIWebIndex can be checked against a published IP range file or a forward-confirmed reverse DNS lookup.
Yes. The operator documents that AIWebIndex reads robots.txt and follows Disallow rules for its own token. Rules take effect on the next fetch of the file, which can be hours or days after you publish them.
Lyrenth publishes its crawl IPs as JSON (gptbot.json format). Reverse DNS resolves under lyrenth.com and forward-resolves back to the same IP. A Web Bot Auth (RFC 9421) key directory is also published at api.lyrenth.com/.well-known/http-message-signatures-directory. All 22 published addresses forward-confirmed on 2026-08-24. The reference is published at https://lyrenth.com/bot/ip-ranges.json.
Add these user-agent tokens (AIWebIndex, AIWebIndex-Agent) to robots.txt with Disallow: /. If the crawler ignores robots.txt, match the user agent at your CDN or reverse proxy and return 403 there instead.
Blocking AIWebIndex takes the site out of the index behind its answers, so it stops being cited there. If referral traffic from AI answers matters to you, block the training crawler instead and leave this one alone.