AI Crawlability Lens · Free Chrome extension
See Any Page the Way GPTBot Sees It
AI Crawlability Lens is a free GPTBot checker for Chrome: it fetches the live page you are on as GPTBot, PerplexityBot and ClaudeBot, raw and unrendered, and shows exactly which content never reaches them.
- Version 1.0.0
- Updated 4 Aug 2026
- Free · 100% local
Why a page can rank in Google and still be invisible to AI crawlers
Googlebot renders your JavaScript. GPTBot, PerplexityBot and ClaudeBot mostly do not. They read only the raw HTML your server returns, so if your content is drawn by JavaScript after load, they see an empty shell. A page can rank in Google, pass every SEO audit and still be missing from AI answers. AI Crawlability Lens is a search simulator extension that shows you that difference on the live page, using each crawler's real User-Agent.
How the AI crawler checker works
Everything starts from the page you are already viewing in Chrome.
- Open the page and click Check. Site access is requested for that site at that moment.
- The page is fetched nine times: once as each major search and AI crawler, and once with an ordinary browser User-Agent as a baseline.
- Each raw response is compared with what you actually see on screen.
- Your robots.txt is evaluated for every crawler.
- You get a score, a verdict and an on-page map of what each crawler misses.
What AI Crawlability Lens checks
The checks separate the different reasons a crawler might not see your content, because each one has a different fix.
- Per-crawler visibility: how much of the on-screen content exists in the raw HTML each crawler receives
- robots.txt rules with proper group precedence, including AI-usage tokens that are separate from the search crawler they appear related to
- Server-level blocking, reported separately from a low score
- Cloaking: whether your infrastructure serves different HTML to different crawlers
- Framework-aware advice matched to the stack the page is built on
What you get from each check
Each crawler gets a visibility score out of 100, with headings weighted double, and a plain-English verdict with its reasoning: blocked in robots.txt, refused by your server, or simply absent from the HTML. An on-page overlay outlines every invisible block in red; switch crawlers and the map redraws. The overlay draws over the page and never modifies it. You can also save a self-contained HTML report for a developer, one file with no external references that makes no network requests when opened, and the extension keeps a local history of your last 20 checks.
Privacy: 100% local, no account, no server
Everything runs locally. There is no account, no server and no telemetry. The only requests made are to the page you are on and its robots.txt, and only when you click Check. Site access is requested per site when you click, never for all sites at install, and you can revoke it whenever you like. No page content is ever stored or transmitted.
Honest limits of seeing a page as GPTBot
Requests come from your own IP address, not from a crawler's published address range. The robots.txt findings and whether your content exists in the raw HTML are exact. Treat a server refusal as a lead to confirm in your access logs, not a conclusion.
About AI Crawlability Lens
Frequently asked questions
How can I see my page as GPTBot sees it?
Install AI Crawlability Lens, open the page and click Check. The extension fetches the page with GPTBot's real User-Agent and shows the raw, unrendered HTML result. An overlay outlines in red every block you can see on screen that GPTBot does not receive.
Does GPTBot render JavaScript?
GPTBot, PerplexityBot and ClaudeBot mostly do not render JavaScript, unlike Googlebot. They read the raw HTML your server returns, so content drawn by JavaScript after load can be invisible to them. AI Crawlability Lens shows which parts of your page fall into that gap.
Can allowing a search crawler in robots.txt still block AI answers?
Yes. Some AI-usage tokens are separate from the search crawler they appear related to, so allowing a search engine to crawl your site does not always allow it to use your content in AI answers. AI Crawlability Lens evaluates every crawler against your rules with proper group precedence.
Is a server refusal in the report definitely a block?
Not always. Requests come from your own IP address, not a crawler's published range, so a server refusal is a lead to confirm in your access logs. The robots.txt findings and whether content exists in the raw HTML are exact.
Does AI Crawlability Lens send my pages to a server?
No. Everything runs locally, with no account, no server and no telemetry. The only requests go to the page you are on and its robots.txt, and only when you click Check. No page content is ever stored or transmitted.
Can I share the results with a developer?
Yes. The extension produces a self-contained HTML report: one file with no external references that makes no network requests when opened. Alongside it, the framework-aware advice is matched to the stack the page is built on, so a developer gets the specific thing to change rather than generic guidance.