GuidePublished: June 30, 2026 · Last updated: June 30, 2026 · ~5 min read
Mihir Naik — Senior PM (AI) at seoClarity

AI Search Crawlers: GPTBot, ClaudeBot, PerplexityBot & the Bots Reading Your Site

Before you can decide what to allow or block, you need to know who’s actually crawling—and why. AI crawlers fall into jobs that look identical in your logs but mean very different things for visibility: training, retrieval, and user-triggered fetches. This reference maps the major bots to their job, user-agent, and how to verify them, so you can make deliberate choices instead of inheriting defaults.
Executive summary
Not every AI crawler affects whether you show up in AI answers. Training crawlers feed models; retrieval crawlers index and fetch live content to build and cite answers; user-triggered fetchers grab a single page because a human asked. Confusing them is how sites accidentally disappear from AI search.
  • Retrieval bots (OAI-SearchBot, PerplexityBot, and the systems behind Google’s AI surfaces) are the ones that decide your citation visibility.
  • Training bots (GPTBot, CCBot, Google-Extended) are a separate, content-licensing decision—blocking them does not remove you from AI answers.
  • User-agents change and are trivially spoofed—verify with each operator’s published IP list or reverse DNS, not the UA string alone.

Training, retrieval, and user-triggered: three jobs, not two

Every AI crawler is doing one of three jobs. Telling them apart is the most important thing on this page.
Training crawlers
  • Collect content to train or fine-tune large language models.
  • Blocking them is a legitimate intellectual-property choice and has no direct effect on whether you’re cited in live AI answers.
  • Examples: GPTBot (OpenAI), CCBot (Common Crawl, which feeds many models), and the directive-only tokens Google-Extended and Applebot-Extended.
Retrieval / search crawlers
  • Index and fetch live content to build and cite answers—the path to being cited.
  • Blocking them directly removes you from those products’ AI answers.
  • Examples: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), and Googlebot, which still governs access for Google’s AI Overviews.
User-triggered fetchers
  • Fetch a single URL in real time because a person asked the assistant to read it—neither bulk training nor scheduled indexing.
  • Examples: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher.
Rule of thumb: if a bot’s job is to answer a user right now, treat blocking it as a revenue decision—not a privacy default.
Google is the cleanest illustration of why the distinction matters. Google states that Google-Extended doesn’t affect a site’s inclusion or ranking in Search—it only opts your content out of Gemini/Vertex training. The corollary catches teams out: you can’t opt out of in-Search AI Overviews via Google-Extended either; AI Overviews eligibility follows Googlebot indexing and snippet settings.
For leadership
When someone proposes ‘blocking AI bots,’ ask which job they mean. Opting out of training is reasonable; opting out of retrieval means opting out of AI-search demand.
Sources & further reading

The crawlers worth knowing by name

A working roster of the agents most likely to show up in your logs, grouped by operator. User-agent tokens and IP ranges change often—confirm the current values against each operator’s official docs before writing rules.

OpenAI

  • GPTBot — training crawler (token GPTBot/1.2 at time of writing). Controlled in robots.txt via the GPTBot user-agent.
  • OAI-SearchBot — retrieval crawler for ChatGPT search results and citations.
  • ChatGPT-User — user-triggered fetch when a person asks ChatGPT to read a URL. All three are documented and independently controllable.

Anthropic

  • ClaudeBot — training crawler.
  • Claude-SearchBot — indexing/retrieval for Claude’s search.
  • Claude-User — user-triggered fetch. Anthropic states all three honor robots.txt, but publishes no IP ranges—so robots.txt, not IP allowlisting, is the supported control.

Perplexity, Google, Microsoft, Apple & others

  • PerplexityBot — Perplexity’s retrieval crawler; Perplexity-User is its user-triggered fetcher.
  • Googlebot — still the access path for Google’s AI Overviews and AI Mode. Google-Extended only governs Gemini training, not ranking or retrieval.
  • Bingbot — powers Microsoft Copilot answers via the Bing index (renders JS with the Edge engine).
  • Applebot — Apple’s search crawler; Applebot-Extended is a directive-only training opt-out that ‘does not crawl webpages.’
  • Amazonbot, Meta-ExternalAgent (training) / Meta-ExternalFetcher (user), ByteSpider, CCBot — additional crawlers worth identifying so you can decide on each.
Sources & further reading

How to see which bots actually reach you—and prove who they are

The roster above is theory until you check your own logs. Your server or CDN access logs are the ground truth for who’s crawling and what they receive. But a user-agent string is trivially spoofed—Google itself notes the Googlebot UA is often faked—so identity has to be verified by IP or reverse DNS.
From your logs, answer three questions per bot:
  1. Is it hitting you at all? Filter by user-agent for each crawler above.
  2. What status code does it get? 200 means access; 403/429/503 means something is turning it away.
  3. Is the traffic real? Verify against the operator’s published IP list or reverse DNS before trusting—or blocking—it.
Where to verify (official IP lists / methods):
  • OpenAI — openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json.
  • Perplexity — perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json (match UA and IP).
  • Google — reverse DNS to googlebot.com / google.com / googleusercontent.com, forward-confirmed.
  • Bing — reverse DNS ending in search.msn.com. Amazon — developer.amazon.com/amazonbot/ip-addresses. Apple — *.applebot.apple.com.
  • Anthropic — no published IP ranges; rely on robots.txt and traffic patterns rather than IP allowlisting.
A retrieval bot consistently receiving 403s is a direct, measurable loss of AI visibility—and one of the fastest wins you’ll find.
Sources & further reading
Part 2 of 6
A part of Why AI Search Engines Can’t See Your Website — and How to Fix It.
  1. Why AI Search Engines Can’t See Your Website — and How to Fix It
  2. AI Search Crawlers: GPTBot, ClaudeBot, PerplexityBot & the Bots Reading Your Site · you’re here
  3. How to Configure robots.txt for AI Crawlers (Without Going Invisible)
  4. Do AI Crawlers Render JavaScript? Rendering & Content Extractability
  5. How to Audit Your Site for AI Search Crawlability (Step-by-Step)
  6. Fixing AI Crawlability by Platform: Next.js, WordPress, Headless & CDN
← Overviewrobots.txt & CDN →
What’s next
Becoming visible in AI search is a technical problem with a revenue outcome. If you'd rather have it diagnosed and fixed, here's how I can help.
Mihir Naik
About the author
Mihir Naik — Senior Product Manager (AI) at seoClarity, building Clarity ArcAI. Born in Surat, India; based in Toronto. In SEO since 2011. Available for consulting.
Read full bio →