GuidePublished: June 30, 2026 · Last updated: June 30, 2026 · ~6 min read
Mihir Naik — Senior PM (AI) at seoClarity

How to Configure robots.txt for AI Crawlers (Without Going Invisible)

robots.txt is the first gate every compliant AI crawler checks—and the easiest place to accidentally erase yourself from AI answers. This part covers how it actually resolves rules per bot, the blanket-rule mistakes that quietly cost visibility, and the CDN/WAF layer that can block AI crawlers even when your robots.txt says ‘come on in.’
Executive summary
Two layers decide whether a crawler reaches your content: robots.txt (a request compliant bots honor) and your CDN/WAF (a hard gate that can block bots regardless of robots.txt). Most accidental AI-invisibility lives in one of these two places.
  • Configure robots.txt per user-agent so you can allow retrieval bots while making a separate call on training bots.
  • A single broad Disallow—or a staging rule shipped to production—can remove you from every AI answer.
  • Even a perfect robots.txt won’t help if edge bot-mitigation is returning 403s to AI user-agents.

How robots.txt governs AI crawlers

robots.txt lives at the root of your domain and lists rules grouped by user-agent. It’s standardized as RFC 9309, the Robots Exclusion Protocol. Two rules from that spec surprise people and cause most misconfigurations:
  • A crawler obeys only the single most specific matching user-agent group—not a union of groups. If a group names the bot (e.g. GPTBot), the wildcard * group is ignored entirely for that bot.
  • Within a group, the most specific path rule wins (the one matching the most characters); on an exact tie, Allow beats Disallow. If no rule matches, the URL is allowed by default.
Anatomy
# https://example.com/robots.txt
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Because named groups override the wildcard, a bot listed in its own group ignores your `User-agent: *` rules completely. Don’t assume a global rule also covers a specifically-named bot.
One more crucial limit: robots.txt is a published preference, not access control. Compliant bots honor it; bad actors ignore it. It also isn’t a de-indexing tool (see below).
Sources & further reading

Per-bot configuration patterns

Because training and retrieval are different decisions, your robots.txt should treat them differently. Three common, deliberate postures:
Opt out of training, stay visible in AI answers
# OpenAI: block training, allow search + user fetch
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /

# Anthropic: block training, allow search + user fetch
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /

# Google generative-AI training opt-out (control token, not a crawler)
User-agent: Google-Extended
Disallow: /
Fully open (maximize AI visibility)
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Sources & further reading

The accidental block

Most lost AI visibility isn’t a policy choice—it’s a leftover. The usual suspects:
  • A site-wide Disallow: / that shipped from a staging environment to production.
  • A blanket ‘block all bots’ rule added during a scraping scare that also catches retrieval crawlers.
  • Disallowing the directories where your real content lives (e.g. /blog, /resources) while leaving marketing pages open.
  • A noindex or nosnippet directive that suppresses snippet generation—and with it, AI-snippet eligibility.
The most expensive two lines on the web
User-agent: *
Disallow: /
Two directive traps worth calling out explicitly. First, robots.txt Disallow does not de-index a page—Google can still list a disallowed URL (without a snippet) if it’s linked elsewhere, and worse, if you Disallow a page Google can never see a noindex on it. Second, Google’s nosnippet and max-snippet directives also limit whether content can be used as input for AI Overviews and AI Mode—so a well-meant snippet control can quietly remove you from AI answers.
After any deploy that touches infrastructure, re-check robots.txt in production. It’s a 30-second check that prevents a category of silent, total AI invisibility.
For leadership
Make ‘verify production robots.txt’ a release checklist item. The failure mode is invisible in dashboards and can persist for months before anyone notices the missing demand.
Sources & further reading

When robots.txt isn’t the blocker: CDN & WAF

You can have a flawless robots.txt and still be invisible. CDNs and web application firewalls sit in front of your origin and decide about automated traffic before robots.txt is ever read. Cloudflare, which sits in front of a large share of the web, now ships several AI-crawler controls that can deny bots at the edge:
Common hard blocks:
  • The one-click ‘Block AI bots’ toggle in Super Bot Fight Mode (introduced Sept 2024), which returns 403 to known AI crawlers.
  • As of July 2025, Cloudflare blocks AI crawlers by default for new domains—so a bot you assume is allowed may be denied out of the box.
  • Bot-fight / managed-challenge modes that serve a JavaScript or CAPTCHA challenge a crawler can’t solve.
  • Rate limiting that trips on a crawler’s normal request pace and returns 429.
The fix is to explicitly allow the retrieval crawlers you’ve decided to welcome. Cloudflare’s Verified Bots concept (validated primarily by reverse DNS) and its per-crawler AI Crawl Control (formerly ‘AI Audit’) are the enforcement layer—Cloudflare frames robots.txt as the preference and these tools as the control, and recommends using both.
Sources & further reading

Verify it actually works

Don’t trust the config—test the response a crawler gets. Curling robots.txt validates the file; curling a real content URL with an AI user-agent validates the edge. You need both, because a 200 on robots.txt tells you nothing about whether the WAF blocks the bot on an actual page.
Check the response an AI crawler receives
# 1) Does robots.txt serve to an AI UA?
curl -A "GPTBot/1.2" -sS https://example.com/robots.txt

# 2) Does the EDGE actually let the bot fetch a real page? (watch the status)
curl -A "OAI-SearchBot" -s -o /dev/null -w "%{http_code}\n" https://example.com/your-page
# 200 = reachable · 403 = WAF/challenge block · 429 = rate-limited
Test from outside your network. Internal requests often bypass the very CDN/WAF rules that block crawlers, hiding the problem. And remember UA can be spoofed—for true identity, verify against the operator’s IP list.
Sources & further reading
Part 3 of 6
A part of Why AI Search Engines Can’t See Your Website — and How to Fix It.
  1. Why AI Search Engines Can’t See Your Website — and How to Fix It
  2. AI Search Crawlers: GPTBot, ClaudeBot, PerplexityBot & the Bots Reading Your Site
  3. How to Configure robots.txt for AI Crawlers (Without Going Invisible) · you’re here
  4. Do AI Crawlers Render JavaScript? Rendering & Content Extractability
  5. How to Audit Your Site for AI Search Crawlability (Step-by-Step)
  6. Fixing AI Crawlability by Platform: Next.js, WordPress, Headless & CDN
← The crawlersRendering →
What’s next
Becoming visible in AI search is a technical problem with a revenue outcome. If you'd rather have it diagnosed and fixed, here's how I can help.
Mihir Naik
About the author
Mihir Naik — Senior Product Manager (AI) at seoClarity, building Clarity ArcAI. Born in Surat, India; based in Toronto. In SEO since 2011. Available for consulting.
Read full bio →