AI Crawler Analyzer
Paste your server or CDN access logs and see exactly how GPTBot, ClaudeBot, PerplexityBot and other AI crawlers hit your site - with forward-confirmed reverse-DNS spoof detection for the fake bot traffic impersonating them.
Works with Apache/Nginx/Cloudflare/Vercel logs. Nothing is stored. Up to 20,000 lines.
Paste access logs above to see how AI crawlers hit your site.
Paste raw access logs and the analyzer classifies every line against eighteen known AI and search crawlers, then runs forward-confirmed reverse DNS to separate the real bots from the ones impersonating them. You get per-bot hit counts, the paths each one crawls, a crawl timeline and a table of spoofed IPs. It runs on parsing and DNS alone, nothing is stored, and it is free.
What one pass extracts
Up to 20,000 lines are read per run. Each line is scanned for a client IP, the user-agent string, the requested path and a timestamp in either Apache bracket format or ISO, which is why Apache and Nginx access logs work alongside Cloudflare and Vercel exports without reformatting. Lines matching no known crawler are counted as parsed and then ignored. No third party sees the data: the whole analysis is string parsing plus Node DNS lookups.
- Hits, unique IPs and the top three paths for every crawler seen
- A timeline bucketed by hour when all hits fall on one day, otherwise by day
- The fifteen most-crawled paths across all detected bots
- A suspicious table listing spoofed IPs with the bot they claimed and the host they resolved to
- The full result set, copyable as JSON
Verified, unverified and spoofed
Verification only applies to operators that publish resolvable hosts: OpenAI, Anthropic, Perplexity, Google and Microsoft. For those, an IP is reverse-resolved, and the hostname that comes back is then forward-resolved to check it points at the same address. Match the operator domain in both directions and the IP is verified. Resolve cleanly to something outside the allowlist and it is spoofed. Anything ambiguous is left unresolved rather than accused, and crawlers with no published host range, such as Bytespider, Amazonbot, CCBot and Applebot, always report as unverified. Forty IPs are checked per run.
What GPTBot log analysis is actually for
Two questions make this worth running. First, are the AI crawlers reaching the pages you want quoted, or spending their budget on tag archives, filters and pagination? The top-paths panel answers that in seconds. Second, how much of your apparent AI-bot traffic is genuine? Spoofed hits inflate every internal story you tell about AI referral growth, and they can justify blocking rules aimed at the wrong thing entirely. Run it after a robots.txt change, or whenever a new crawler shows up in your bandwidth graphs.
Frequently asked questions
about the AI Crawler Analyzer
What logs does the AI crawler analyzer accept?
Standard Apache and Nginx access logs, plus Cloudflare and Vercel log exports. Paste up to 20,000 lines; nothing is uploaded or stored beyond your session.
Which AI crawlers can it detect?
GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-User from Anthropic, PerplexityBot, Google-Extended, plus Bytespider, Amazonbot, CCBot, Applebot and others.
How does spoof detection work?
For crawlers that publish verifiable hosts, the tool does a forward-confirmed reverse DNS lookup on each IP. If an IP claims to be GPTBot but does not resolve to an OpenAI host, it is flagged as spoofed - a large share of claimed AI-bot traffic is fake.
Related free tools
pairs well with the AI Crawler Analyzer
Browse all free SEO & AI-search tools, or get the Chrome extensions.
Further readingAI crawlers explained: GPTBot, ClaudeBot and PerplexityBotThe user agents, how to verify them, and what blocking them actually costs.