Free tool · No signup
Enter any URL and we fetch it six times — as Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bingbot, and a regular browser — then compare status codes, robots.txt access, and how much text each crawler can actually read.
Free · Unlimited checks · No signup required
Step 1
Enter a URL
Any public page — yours or a competitor's.
Step 2
Six fetches in parallel
We request the page as each crawler, with its real user-agent string, plus your robots.txt.
Step 3
Side-by-side comparison
Status code, robots.txt verdict, extractable text, and title — one row per crawler.
Step 4
Read the verdicts
Plain-language callouts flag blocked bots and JavaScript-rendered content gaps.
Why it matters
ChatGPT, Claude, and Perplexity build answers from pages their crawlers can actually read. A robots.txt disallow, a firewall rule, or bot-protection middleware that returns 403 to GPTBot silently removes you from every answer those engines generate. The page looks fine in your browser, so nobody notices — until a competitor gets the citation instead.
Most AI crawlers don't execute JavaScript. If your content renders client-side, a crawler may receive a near-empty shell while your browser shows a full article. Comparing the text volume each user-agent receives is the fastest way to detect this gap: when bots get under 60% of what the browser gets, your content is effectively invisible to AI engines.
Some sites deliberately block AI training crawlers, and that's a legitimate choice. The problem is accidental blocking: a CDN bot-fight mode, an overzealous WAF rule, or a robots.txt line copied from a template. Running this simulation takes seconds and tells you exactly which crawlers are blocked and at which layer — robots.txt or the server itself.
With Meev
Crawl access is step one. Meev publishes AI-extraction-optimized articles to your domain on autopilot and then monitors every major AI search surface — ChatGPT, Claude, Gemini, Perplexity, Google AI Overviews — to show where your brand appears in real answers.
Six in total: Googlebot (Google Search), GPTBot (the crawler behind ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), Bingbot (Bing), and a regular desktop browser as the control. Each fetch uses that crawler's real user-agent string so server-side bot rules respond exactly as they would in production.
Your robots.txt file contains a rule that disallows that crawler from fetching the page's path. Well-behaved crawlers like GPTBot and ClaudeBot respect these rules, so a disallow line effectively removes your content from those AI engines. We check both crawler-specific groups and the wildcard (*) group.
Most AI crawlers don't run JavaScript. If your page builds its content in the browser — common with single-page-app frameworks — crawlers receive only the initial HTML shell. Server-side rendering or static generation fixes this: the full content ships in the first response, so every crawler reads what your visitors read.
Your server, CDN, or bot-protection layer is filtering by user-agent or bot fingerprint. Common culprits are CDN bot-fight modes, WAF rules, and security plugins with default-deny bot lists. If you want AI engines to cite you, add explicit allow rules for the crawlers you trust.
It depends on your goals. Blocking GPTBot or ClaudeBot keeps your content out of training data and AI answers — which protects exclusivity but forfeits citations and the referral traffic AI engines increasingly send. Most businesses competing for visibility want AI crawlers allowed; publishers selling exclusive content sometimes don't.
No — Search Console only tests Googlebot. This tool adds the AI crawlers that Search Console doesn't cover: GPTBot, ClaudeBot, and PerplexityBot. A page can be perfectly indexed in Google while being completely invisible to ChatGPT and Perplexity, and only a multi-crawler comparison reveals that.
Meev tracks your visibility across every major AI search surface and publishes quality-gated content that earns citations — automatically.
Card required, no charge until day 8. Cancel anytime.
AI Search
LLMs.txt Generator
Generate an llms.txt file so AI engines index your site correctly
AI Search
AI SEO Audit
Audit any page for AI-search readiness in 30 seconds
AI Search
Query Fan-Out Generator
Turn one query into 25+ AI-search variations, grouped by intent
AI Search
Page Token Inspector
See your page the way an AI engine reads it — token by token
Content & Writing
Blog Post Ideas Generator
15 SEO-ready blog post ideas for your niche in seconds
Content & Writing
Content Brief Generator
A full SEO content brief — outline, keywords, FAQs — in seconds