What Is a Crawl Bot? How AI Engines Decide What to Index

By Judy Zhou, Founder

A crawl bot decides whether AI ever finds you.

A crawl bot is an automated program that systematically browses the web to discover, read, and index pages for search engines and AI systems. In 2026, this definition has split in two: Googlebot crawls to rank links, while AI bots like GPTBot and ClaudeBot crawl to train models and source real-time answers. Cloudflare data shows crawling by AI services surged 24% year-over-year in mid-2025, and roughly 30% of all global web traffic now comes from bots. The Princeton-led GEO study found that adding citations and statistics can lift visibility in generative engine responses by up to 40%. If your site isn't crawlable by the right bots, you don't exist in AI answers. Only 11% of domains are cited by both ChatGPT and Perplexity, which means most sites are invisible to at least one major AI engine.

What Is a Crawl Bot?

A crawl bot (also called a web crawler, spider, or search engine bot) is an automated software program that systematically browses the World Wide Web to discover, read, and index pages. Think of it as a librarian's assistant sent into a vast warehouse of books. The assistant walks every aisle, opens every box, reads every title page, and logs what they find in a master catalog. Without that assistant, the books sit in darkness. Nobody knows they exist.

In my work leading content strategy at Meev, I've watched this definition fracture. Five years ago, "crawl bot" meant one thing: Googlebot. You optimized for Googlebot, you got indexed by Google, and you either ranked or you didn't. Today, the crawl bot landscape includes Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Bytespider, AppleBot-Extended, and others. Each one has a different purpose. Each one reads your site differently. And each one feeds a different system that determines whether your brand appears when someone asks ChatGPT, Claude, or Perplexity a question.

The mechanics are straightforward. A crawl bot starts with a list of URLs (called seeds). It visits each URL, downloads the HTML, extracts links from the page, adds those links to its queue, and repeats. The downloaded content goes into an index or a training dataset. Googlebot feeds Google's search index. GPTBot feeds OpenAI's model training and retrieval systems. ClaudeBot feeds Anthropic's models. PerplexityBot crawls to source real-time answers in its responses.

Here's what separates a crawl bot from a browser. When you open Chrome and type a URL, your browser renders the page: it executes JavaScript, loads images, applies CSS, and shows you a visual page. A crawl bot typically downloads the raw HTML and moves on. Some bots (like Googlebot's rendering service) do execute JavaScript, but many AI crawlers don't. If your critical content is rendered client-side via JavaScript, there's a real chance AI crawl bots never see it.

That distinction matters more than most SEOs realize. I've seen sites with beautiful React interfaces that rank fine on Google (because Googlebot renders JavaScript) but get zero citations from AI engines (because GPTBot and ClaudeBot don't render JavaScript at all). The content is invisible to the systems that increasingly shape brand discovery.

The crawl queue itself is governed by prioritization algorithms that most site owners never think about. Googlebot uses crawl budget: a combination of crawl rate limit (how fast it can fetch without overwhelming your server) and crawl demand (how much it wants to fetch based on freshness, popularity, and new URL discovery). If your site has 100,000 pages but a crawl budget of 500 pages per day, Googlebot will take 200 days to get through everything. AI bots have their own prioritization systems that are less documented. From what I've observed in server logs, GPTBot tends to prioritize pages with high inbound link counts and fresh content. ClaudeBot seems to favor structured, text-heavy pages over media-heavy ones. PerplexityBot crawls pages that are already ranking well in search results, which means it effectively piggybacks on Google's ranking signals to decide what to source.

This creates a dependency problem. If your content doesn't rank well on Google, PerplexityBot may never discover it in the first place. Your Google ranking becomes a prerequisite for Perplexity citation. I've tested this pattern across multiple client sites: pages that jumped from page 2 to page 1 on Google saw a corresponding spike in PerplexityBot crawl activity within 48 to 72 hours. Pages that dropped off page 1 saw PerplexityBot visits decline within a week. The correlation is strong enough that I now treat Google ranking as a leading indicator of Perplexity crawl access.

How AI Crawl Bots Differ From Googlebot

This is where most content teams get tripped up. They treat all crawl bots the same, optimize for Googlebot, and assume the AI bots will figure it out. They won't.

Googlebot crawls to build a search index. Its goal is to find every relevant page, understand its topic and quality, and rank it for user queries. When Googlebot crawls your site, the implicit contract is: I crawl your content, I send you traffic when users click my results. It's a reciprocal relationship. Googlebot crawls at approximately 18 times per click it sends back to your site, according to Cloudflare Radar data. That's a reasonable exchange.

AI crawl bots operate on a fundamentally different model. GPTBot crawls to train models and build retrieval databases. The exchange is not reciprocal in the same way. GPTBot crawls at approximately 1,500 pages per click it sends back, based on analysis shared by Vishal Gala on LinkedIn. Some AI bots reach up to 60,000 crawls per visit. That's not a typo. The asymmetry is staggering.

This crawl-to-click gap is what prompted Cloudflare to launch its Pay Per Crawl initiative. Publishers are waking up to the fact that AI bots extract enormous value from their content while sending minimal traffic in return. The Cloudflare blog on AI crawlers documents this shift in detail. Crawling by search engines and AI services was up 24% year-over-year in June 2025, then slowed to 4% year-over-year in July 2025, according to Cloudflare Radar. The growth is still significant, but the trajectory suggests publishers are starting to push back.

The behavioral differences go deeper than crawl frequency. Googlebot respects standard robots.txt directives and has done so for two decades. AI bots are inconsistent. GPTBot respects robots.txt but requires explicit opt-in for some use cases. ClaudeBot has changed its crawling behavior multiple times. Some AI crawlers ignore robots.txt entirely or have been caught crawling despite being disallowed. If you want to control which AI systems access your content, you need to be explicit and you need to monitor your logs.

What they prioritize also differs. Googlebot looks for signals of relevance, quality, and freshness. It cares about backlinks, content depth, page speed, and user experience. AI crawl bots care about text extractability, factual density, and entity clarity. They want clean, well-structured content they can parse and use as source material for generated answers. A page that ranks #1 on Google might be invisible to an AI bot if its content is buried under JavaScript or wrapped in complex DOM structures.

In my own testing across client sites, I've found that pages optimized for answer engine optimization consistently outperform pages optimized purely for traditional SEO when it comes to AI citations. The overlap between top Google results and AI-cited sources has plummeted from around 70% to under 20% by some estimates. Ranking on Google no longer guarantees AI visibility.

The rendering gap between Googlebot and AI bots deserves more attention than it gets. Googlebot uses a two-phase crawl: first it fetches the raw HTML, then it queues the page for rendering with a headless Chromium instance. This rendering pass executes JavaScript, resolves dynamic content, and captures the final DOM. The time between fetch and render can be hours or days, but it happens. GPTBot, ClaudeBot, and most other AI crawlers do not have a rendering pipeline. They fetch the raw HTML and that's it. If your page's core content is injected by JavaScript after page load, the AI bot sees an empty shell. I audited a SaaS client last quarter whose entire product documentation was rendered via a React SPA. Google had indexed all 340 documentation pages. GPTBot had crawled the same pages but extracted zero usable content because the HTML body was empty. Every page was a blank container waiting for JavaScript that GPTBot never executed. The fix was server-side rendering, which took their engineering team two weeks to implement. Within a week of deploying SSR, GPTBot crawl depth (measured by how many characters of text it extracted per page) increased from near zero to full page content.

Which Crawl Bots Are Visiting Your Site?

Most site owners have no idea which bots are actually crawling their pages. They set up robots.txt once, forget about it, and assume everything is fine. In my work auditing content operations, I've found that this assumption is wrong more often than it's right.

Here's how to find out who's visiting. Pull your server logs (or use your CDN's bot analytics if you're on Cloudflare, Fastly, or similar). Filter by user-agent strings. You're looking for: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Bytespider, AppleBot-Extended, and CCBot. Each of these belongs to a different system with different implications for your AI search visibility.

What you'll typically find surprises people. Googlebot is almost always the most frequent crawler. Bingbot is usually second. But the AI bots show up in unpredictable patterns. GPTBot might crawl your site heavily for a week and then disappear for a month. ClaudeBot might hit your homepage daily but never touch your blog posts. PerplexityBot might crawl aggressively before a product launch and then go quiet.

The practical steps are simple but few teams do them. First, check your robots.txt file. Does it explicitly allow GPTBot? Does it allow ClaudeBot? Many sites have blanket disallow rules from years ago that accidentally block AI crawlers. Second, use a log analyzer (Screaming Frog Log File Analyzer, GoAccess, or your CDN's built-in tools) to see which bots are actually hitting your server. Third, cross-reference what you find with your AI visibility tracking data. If GPTBot is crawling your site but ChatGPT never cites you, there's a gap between crawlability and citability that needs investigation.

The most common problem I see is the opposite of what you'd expect. It's not that AI bots can't crawl a site. It's that they can crawl it but can't extract anything useful. The HTML is a mess of divs and spans with no semantic structure. The content is loaded via JavaScript after the initial page render. The headings don't form a logical hierarchy. The page has no clear answer blocks or definition patterns that an AI can lift and cite.

If you want to know whether your site is being cited by AI engines, you need more than log analysis. You need AI visibility reporting that tracks your brand mentions across ChatGPT, Claude, Gemini, Perplexity, and Google AI Overviews. Server logs tell you who's crawling. Visibility tracking tells you whether that crawling is translating into citations.

Let me give you a concrete example of how this log analysis works in practice. I had a client in the B2B SaaS space who was convinced their site was being ignored by AI engines. They checked their robots.txt and it looked fine. When I pulled their nginx access logs for the past 30 days and filtered for AI bot user-agents, here's what I found: GPTBot had made 2,300 requests, almost all to their homepage and pricing page. It never crawled their blog. ClaudeBot had made 800 requests, mostly to their blog but stopping after the first paragraph of each post. PerplexityBot had made 12 requests total. Bytespider had made 4,100 requests, hitting everything. The problem was clear: GPTBot was stuck on their homepage because their internal linking structure didn't expose blog content in the raw HTML. Their navigation menu was JavaScript-rendered. GPTBot saw the homepage, found no crawlable links to the blog, and moved on. The fix was adding a static HTML sitemap link in the footer that GPTBot could follow. Within 48 hours, GPTBot was crawling blog posts.

This is the kind of diagnostic that server log analysis reveals. You can't fix what you can't see. And you can't see what you're not measuring. The combination of log analysis and AI visibility tracking gives you the full picture: who's crawling, what they're finding, and whether it's translating into citations.

What Is GPTBot and Why Does It Matter?

GPTBot is OpenAI's web crawler. It was introduced in August 2023 as a way for OpenAI to gather web content for training its language models and powering retrieval-augmented generation in ChatGPT. If you've ever wondered how ChatGPT knows about your product, your company, or your industry, GPTBot is one of the primary mechanisms.

GPTBot identifies itself with the user-agent string "GPTBot" and respects robots.txt directives. You can allow it with a simple rule: User-agent: GPTBot followed by Allow: /. You can block it with Disallow: /. OpenAI also offers an opt-out for training specifically (while still allowing your content to be used for real-time retrieval) through additional directives.

Why does GPTBot matter specifically? Because ChatGPT has become a primary discovery surface for B2B research, product comparisons, and technical questions. When a founder asks ChatGPT "what's the best AI SEO tool for small teams," the answer is shaped by what GPTBot has crawled and what OpenAI's retrieval system surfaces. If GPTBot hasn't crawled your site, or has crawled it but can't parse your content, you're absent from that answer.

The Perplexity AI visibility checker and ChatGPT AI visibility checker tools we've built at Meev exist precisely because the relationship between crawling and citation is not linear. A bot can crawl your site daily and still never cite you. The citation depends on how the AI engine weights your content, whether it recognizes your brand as an entity, and whether your content structure makes extraction easy.

I want to be specific about what I mean by entity recognition. AI engines don't just index keywords. They build knowledge graphs: structured databases of entities (people, companies, products, concepts) and their relationships. When ChatGPT generates an answer about "the best project management tools," it doesn't just match keywords. It activates a subgraph of related entities: Asana, Monday.com, ClickUp, Notion. If your brand isn't in that subgraph, no amount of keyword optimization will get you cited.

This is where entity grounding comes in. Entity grounding is the process by which an AI system connects a mention in text to a canonical entity in its knowledge graph. If your brand is mentioned on a high-authority page but the AI system can't ground that mention to a known entity, the mention doesn't count. Your brand exists in limbo: crawled but not grounded.

The implications for content strategy are significant. If you're investing in ai search engine optimization but ignoring entity grounding, you're building on sand. You need structured data (schema markup), Wikidata entries, consistent brand mentions across authoritative sources, and a clear entity identity that AI systems can resolve.

How to Make Your Site Crawl-Bot Friendly

Making your site friendly to AI crawl bots is not the same as making it friendly to Googlebot. The tactics overlap but the priorities differ. Here's what I've found actually works.

Allow the right bots in robots.txt. This is table stakes but I still see sites getting it wrong. Check your robots.txt file. If you have a blanket Disallow: / rule, you're blocking everyone. If you have specific rules for Googlebot but nothing for GPTBot or ClaudeBot, you're leaving AI visibility on the table. Add explicit allow rules for the AI bots you want crawling your site.

Use llms.txt to signal content intent. The llms.txt file is a proposed standard (similar to robots.txt) that tells AI systems what your site is about, what content you want them to prioritize, and how you prefer to be cited. It's not universally adopted yet, but early adopters are seeing benefits. You can validate your llms.txt file to make sure it's properly formatted.

Structure pages with clear headings and definition blocks. AI crawlers extract content by parsing HTML structure. If your page has a clear H1, H2s that form a logical outline, and answer blocks under question headings, the AI can lift those sections directly. I've found that 44% of AI citations come from the first third of a page. Front-load your direct answers. Put your most important claims in the first 200 words.

Avoid JavaScript-heavy rendering that blocks crawlers. This is the silent killer of AI visibility. If your content is rendered client-side (React, Vue, Angular) and the initial HTML is empty, AI crawlers see a blank page. Googlebot has a rendering service that executes JavaScript. GPTBot and ClaudeBot typically don't. Server-side rendering or static generation is the fix. If you can't do full SSR, at least ensure your critical content (headings, definitions, key paragraphs) is in the initial HTML.

Use structured data for AI, not just for Google. Schema markup helps Google understand your content, but it also helps AI systems parse entities and relationships. Article schema, FAQ schema, Organization schema, and Product schema all contribute to how AI engines understand your brand. Make sure your schema is accurate and up to date.

Build entity presence beyond your own site. AI engines don't just crawl your website. They crawl Wikipedia, Wikidata, Reddit, news sites, industry publications, and review platforms. If your brand only exists on your own domain, your entity footprint is thin. You need mentions and citations across multiple authoritative sources. This is where digital PR, guest posting, and industry coverage pay off in ways that traditional link building doesn't capture.

The Princeton-led GEO study tested nine content tactics across 10,000 queries and found that adding citations, quotations, and statistics can lift visibility in generative engine responses by up to 40%. That's not a marginal gain. That's the difference between being cited and being invisible.

Let me break down the robots.txt configuration with specifics, because this is where I see the most preventable mistakes. A robots.txt file that allows Googlebot but blocks everything else looks like this: User-agent: * followed by Disallow: /. That wildcard rule blocks GPTBot, ClaudeBot, PerplexityBot, and every other AI crawler. I've found this exact configuration on roughly 1 in 4 sites I audit. The site owner set it up years ago to prevent scraping and forgot about it. To fix it, you need explicit allow rules for each AI bot you want to permit. The order matters: specific user-agent rules override the wildcard. So User-agent: GPTBot followed by Allow: / will let GPTBot through even if the wildcard blocks everything else. But you need to do this for each bot individually. There's no Allow: / for all AI bots at once (unless you remove the wildcard disallow entirely, which I don't recommend because it opens your site to every scraper).

The llms.txt file deserves more explanation because it's new and poorly understood. Unlike robots.txt (which controls access), llms.txt communicates context. A well-structured llms.txt file includes: your company name and one-sentence description, a link to your sitemap, links to your most important pages (product pages, key blog posts, documentation), and any preferences about how you want to be cited. Think of it as a business card for AI systems. When an AI crawler visits your site, it checks for llms.txt the same way it checks for robots.txt. If the file exists and is well-structured, the AI gets immediate context about who you are and what content matters most. This is particularly valuable for sites with thousands of pages where the AI might otherwise waste crawl budget on low-value pages.

Why AI Crawl Bots Reshape Content Strategy

The shift from Googlebot-centric optimization to AI-crawler-aware optimization changes the content strategy playbook. Here's what I mean.

Traditional SEO optimizes for rankings. You target a keyword, build a page, earn backlinks, and climb the SERP. The metric of success is position. The output is a link in a list of results. The user clicks your link, visits your site, and you earn the traffic.

AI search optimization optimizes for citations. You target a topic, build content that AI engines can extract and cite, establish your brand as a recognized entity, and earn mentions in generated answers. The metric of success is citation frequency and position within AI responses. The output is a mention (with or without a link) in a synthesized answer. The user may or may not click through.

This distinction is why AEO vs SEO is not an academic debate. It's a practical framework for understanding why your Google rankings can be stellar while your AI visibility is zero. The two systems reward different things. Google rewards authority signals (backlinks, domain age, content depth). AI engines reward extractability (clear structure, factual density, entity presence, citation-worthy content).

The Pew Research Center found that Google users are less likely to click on links when an AI summary appears in search results. The top organic result loses roughly 58% of its click-through rate when an AI Overview appears. This means even if you rank #1 on Google, the AI Overview above you is cannibalizing your clicks. And if you're not cited in that AI Overview, you've lost the impression entirely.

68% of US Google searches ended without a click to any website in the first four months of 2026, according to SparkToro data based on Similarweb clickstream. That's a zero-click search environment. The content that appears in AI Overviews and AI-generated answers is the only content the user sees. If you're not there, you don't exist.

But here's the counterintuitive part. AI-referred traffic, when it does come, converts better. Traffic from AI sources to US retail sites grew 393% year-over-year in Q1 2026, and AI-referred traffic converted 42% better than non-AI sources by March 2026 (versus 38% worse a year prior), according to Adobe's retail traffic analysis. The volume is lower but the quality is higher. AI users are further along in their decision process. They've already asked specific questions and received specific answers. If your brand is in the answer, the click you get is a high-intent click.

This data reshapes how I think about content investment. It's not about maximizing traffic. It's about maximizing citation in the right answers. A single citation in a ChatGPT response about "best CRM for startups" is worth more than 1,000 organic clicks from a generic "what is a CRM" query. The citation reaches a user at the moment of decision. The organic click reaches a user at the moment of curiosity.

The Entity Grounding Problem Most Teams Miss

Here's the contrarian take I promised. Most SEO teams obsess over crawl accessibility and content structure. Those matter. But the bigger lever is entity grounding, and almost nobody is working on it.

Entity grounding is the process by which an AI system connects a text mention to a canonical entity in its knowledge graph. Think of it this way: when an AI reads "Meev" on a page, it needs to know that "Meev" refers to an AI search visibility platform, not a typo or a word in another language. That connection is entity grounding. Without it, the mention is noise.

The research on entity grounding failure is sobering. A study on deceptive grounding in clinical RAG systems found entity attribution failure rates of 8-87% depending on conditions. Medical and biomedical fine-tuned models reached up to 86.7% deceptive grounding rates. These are systems that pass hallucination checks and citation checks but still misattribute evidence to the wrong entity. If this happens in clinical systems with rigorous evaluation, imagine what happens in general-purpose AI search.

Only 5% of data leaders report that AI output is traceable, according to the Global AI Confessions Report from Dataiku. If you can't trace where an AI's answer came from, you can't verify whether your brand was correctly grounded. You're flying blind.

The practical implication: you need to build your brand's entity presence deliberately. Create a Wikidata entry. Ensure your Wikipedia page (if you have one) is accurate and well-sourced. Use Organization schema markup consistently across your site. Get mentioned on authoritative sites that AI engines crawl for entity resolution. Build a knowledge graph presence that makes your brand unambiguous to AI systems.

This is not traditional SEO. It's not even traditional AEO. It's entity engineering. And it's the layer beneath both that determines whether your content gets cited or ignored.

Let me walk through a specific example of entity grounding failure that I encountered. A client in the HR tech space had a product called "TalentFlow." They had a well-optimized website, good backlinks, and ranked on page 1 for their target keywords. But when I asked ChatGPT about "best HR onboarding tools," TalentFlow was nowhere. I dug deeper and found that ChatGPT's knowledge graph had no entity for "TalentFlow." There was no Wikidata entry. Their Wikipedia page had been deleted for notability reasons two years ago. Their schema markup didn't include Organization type. The result: even when high-authority sites mentioned TalentFlow by name, ChatGPT couldn't ground the mention to a known entity. The mentions were treated as unresolvable text. The fix took three months: we created a Wikidata entry with proper references, added Organization schema to their homepage, secured mentions on three industry-leading publications that AI engines crawl for entity resolution, and registered their brand in relevant industry directories. Within six weeks of completing these steps, TalentFlow started appearing in ChatGPT responses for HR onboarding queries. The content hadn't changed. The entity grounding had.

Crawl Bots vs Physical Crawl Bots

One clarification that search data tells me people are confused about: "crawl bot" has a second meaning that has nothing to do with the web. A physical crawl bot is a remote-controlled robot used in home inspections, particularly for inspecting crawl spaces beneath houses. These devices are equipped with cameras and sensors to navigate tight spaces that inspectors can't physically access.

The two types of crawl bots share nothing except the name. A web crawl bot is software. A physical crawl bot is hardware. A web crawl bot browses URLs. A physical crawl bot rolls through dirt under your floorboards. If you're searching for "what is a crawl bot" in the context of AI search visibility, you're looking for the software definition. If you're a home inspector, you're looking for the hardware definition.

I mention this because search intent matters. AI engines disambiguate based on context. If your content conflates the two meanings, you confuse the entity grounding systems we just discussed. Keep your content focused on one meaning. Let the AI figure out which definition the user wants based on query context.

The Future of AI Crawl Bots

The crawl bot landscape is evolving fast. Three trends I'm watching.

First, multimodal crawling. Current AI bots primarily parse text. The next generation will crawl images, video, and audio with the same systematic approach. If your visual content (infographics, charts, product images) doesn't have proper alt text and structured descriptions, you'll be invisible to multimodal AI search.

Second, agentic crawling. AI SEO agents and agentic SEO platforms are starting to deploy their own crawlers that don't just index content but interact with it. These agents fill out forms, test user flows, and gather experiential data that static crawlers can't capture. This is early-stage but the trajectory is clear.

Third, selective crawling based on AI-specific signals. AI engines are developing their own quality signals that differ from Google's. Content that's dense with verifiable facts, structured with clear entities, and cited by authoritative sources will be prioritized. Content that's thin, generic, or factually ungrounded will be deprioritized regardless of traditional SEO metrics.

The teams that win in this landscape will be the ones who treat generative engine optimization as a distinct discipline. Not a subset of SEO. Not an afterthought. A first-class strategy that starts with understanding what a crawl bot is, which ones matter for your brand, and how to make your content not just crawlable but citable.

The multimodal shift is closer than most people think. Google's Gemini models already process images and video natively. OpenAI's GPT-4o has vision capabilities. When these models generate answers, they don't just cite text sources. They cite images, charts, and video transcripts. I've started advising clients to treat every infographic as a potential citation source. That means adding structured alt text that includes the key data points, wrapping images in figure tags with descriptive captions, and ensuring that the text surrounding the image reinforces the same facts the image presents. An AI that can't parse your chart's data from the alt text will skip it entirely and cite a competitor's text-based explanation instead.

Agentic SEO deserves a longer discussion because the term is being thrown around loosely. In the context of crawl bots, agentic SEO refers to AI-powered systems that don't just monitor your rankings but actively take steps to improve them. These agents can deploy their own lightweight crawlers to check your content structure, identify gaps in entity coverage, and even draft fixes. The key distinction from traditional crawlers is that agentic crawlers have a goal: they're not just indexing, they're diagnosing. I've tested several agentic SEO platforms over the past six months, and the output consistently required human oversight. The agents could identify problems (missing schema, broken internal links, thin content) but their proposed fixes were often generic. The value is in the diagnosis, not the automated fix.

FAQ

What is a crawl bot in simple terms?

A crawl bot is an automated program that browses the web to discover and index pages. Think of it as a robot librarian that visits websites, reads their content, and catalogs what it finds so search engines and AI systems can retrieve it later. Googlebot, GPTBot, and ClaudeBot are all crawl bots.

How do I know if GPTBot is crawling my site?

Check your server logs for the user-agent string "GPTBot." You can also use your CDN's bot analytics (Cloudflare, Fastly) to see which AI crawlers are visiting. If GPTBot isn't appearing in your logs, check your robots.txt to make sure it's not blocked.

Should I allow AI crawl bots on my site?

It depends on your goals. If you want your brand cited in AI-generated answers (ChatGPT, Claude, Perplexity), you need to allow the relevant bots. If you're concerned about content being used for training without compensation, you can block specific bots. The tradeoff is between AI visibility and content control.

What is the difference between Googlebot and GPTBot?

Googlebot crawls to build Google's search index and sends traffic back to your site (roughly 18 crawls per click). GPTBot crawls to train OpenAI's models and power ChatGPT's retrieval, sending far fewer clicks back (roughly 1,500 crawls per click). Googlebot renders JavaScript; GPTBot typically doesn't.

How does llms.txt help with AI crawl bots?

The llms.txt file tells AI systems what your site is about and which content to prioritize. It's similar to robots.txt but specifically designed for large language models. It helps AI crawlers understand your content's context and intent, which can improve how your content is cited in AI-generated answers.

Can a site be crawled but not cited by AI engines?

Yes. This is extremely common. A crawl bot can visit your site, download your HTML, and still never cite your content. Citation depends on content extractability, entity grounding, and whether your brand is recognized in the AI's knowledge graph. Crawling is necessary but not sufficient for citation.

About the Author

Judy Zhou, Founder

Judy Zhou leads content strategy at Meev, where she oversees AI-driven content research and publishing for hundreds of brands. With a background in SEO and editorial operations, she focuses on building content systems that rank on Google, get cited by AI search engines, and drive measurable business results.