GuidePublished: June 30, 2026 · Last updated: June 30, 2026 · ~5 min read
Mihir Naik — Senior PM (AI) at seoClarity

Fixing AI Crawlability by Platform: Next.js, WordPress, Headless & CDN

Once the audit tells you what’s broken, the fix depends on your stack. This part turns findings into action for the platforms most teams run—Next.js/React, WordPress, headless, and the CDN/edge in front of them—plus how to prioritize and confirm visibility actually recovers.
Executive summary
The blockers are universal; the levers are platform-specific. Fix access at the edge and robots layer first (fastest, highest leverage), then rendering, then extractability—using the controls your stack actually exposes.
  • Next.js/React: move key pages off client-only rendering to SSR/SSG; watch edge middleware.
  • WordPress: it’s server-rendered by default—your risks are SEO/security plugins and caching layers.
  • Headless/Jamstack: pre-render content into HTML; don’t depend on client-side API calls.
  • CDN/edge: allowlist the retrieval bots you’ve chosen so mitigation stops blocking them.

Fix in priority order

Whatever your stack, sequence the work by leverage. Unblocking access is usually a config change with outsized impact; rendering is a build change; extractability is incremental polish.
  1. Access (robots + edge) — config-level, highest leverage, often same-day.
  2. Rendering (SSR/SSG) — get content into the raw HTML for your top templates.
  3. Extractability — semantic markup, un-bury content, replace text-in-images.
Audit the result of a bot request, not just the volume. A thousand GPTBot hits mean nothing if the status code is 403.

Next.js & React SPAs

React itself isn’t the problem—client-only rendering is. A Vite/CRA single-page app ships an empty shell (and often the same title/meta on every route); the fix is to render on the server or at build time.
  • Prefer Server Components / SSR / SSG. In the App Router, generateStaticParams statically generates dynamic route segments at build time.
  • For an existing SPA, migrate the highest-value templates first rather than rewriting everything.
  • Audit edge middleware: the Edge runtime isn’t a full Node.js environment, and redirect/i18n logic in middleware has been reported to misfire for crawlers—make sure it isn’t bouncing bots before they reach the page.
  • Verify dynamic routes actually generate HTML (and aren’t falling back to a client-rendered shell) for the URLs you care about.
Confirm the server returns content, not a shell
curl -s https://example.com/key-page | grep -i "your key phrase"
# present = server-rendered; missing = still client-side
Sources & further reading

WordPress

WordPress renders HTML on the server by default, so rendering is rarely the issue. The risks come from plugins and caching.
  • SEO plugins: Yoast SEO Premium ships a Bot Blocker that toggles robots.txt rules for AI bots like GPTBot, CCBot, and Google-Extended; Rank Math manages AI crawlers via its robots.txt editor. Check these aren’t emitting noindex/nosnippet or blocking bots you want.
  • Security/firewall plugins: Wordfence blocks by custom user-agent pattern (one user-agent per rule, no comma-separated lists) and its rate-limiting can throttle legitimate bots—set thresholds generously.
  • Caching/CDN: make sure cached responses are full HTML and the CDN isn’t bot-gating AI crawlers.
  • Page builders: some load content via JavaScript or lazy-load blocks—verify the served HTML contains the real text.
Sources & further reading

Headless & Jamstack

Headless setups can be excellent for AI visibility—or accidental SPAs—depending on where rendering happens.
  • Pre-render content into HTML at build (SSG) or request time (SSR/ISR); don’t rely on the browser fetching from your CMS API after load.
  • Keep pre-rendered HTML fresh with on-demand ISR: a CMS publish fires a webhook that revalidates just the affected pages—static output, current content, no client fetches.
  • If you use static regeneration, confirm pages are actually built and served as HTML, not deferred to client fetches.
Sources & further reading

CDN & edge

The edge is where the highest-leverage access fixes usually live. If logs show retrieval bots getting 403/429, this is your first stop.
  • Allowlist chosen crawlers via Verified Bots (reverse-DNS/ASN validated), not user-agent alone—UA is spoofable.
  • Review the ‘Block AI bots’ toggle in Super Bot Fight Mode; WAF custom rules run before it, so use a custom allow rule (or per-crawler AI Crawl Control) to permit the bots you want.
  • On the free Bot Fight Mode tier you can’t bypass with custom rules—granular allowlisting needs Super Bot Fight Mode (Pro+).
  • Loosen rate limits that trip on normal crawl pacing; return 200s, not 429s, to legitimate bots.
Sources & further reading

Confirm visibility recovers

A fix isn’t done until the data moves. Close the loop with the same instruments you used to find the problem.
  1. Re-check logs: retrieval bots now receiving 200s where they got blocked (e.g. OAI-SearchBot getting 200 while a training bot you disallowed still gets 403 confirms your rules work).
  2. Re-run the render-diff: your content present in the raw HTML for the fixed templates.
  3. Track AI referral traffic in GA4: filter Traffic acquisition by session source/medium for chatgpt.com, perplexity.ai, gemini.google.com, claude.ai, or build a custom channel-group regex. Native attribution is partial and referrers are often stripped, so pair it with an AI-visibility tracker to catch zero-click citations.
Recovery isn’t instant—crawlers have to revisit and answer indexes have to refresh, which can take days. Fix, verify access, then give it time and re-measure.
Sources & further reading
Part 6 of 6
A part of Why AI Search Engines Can’t See Your Website — and How to Fix It.
  1. Why AI Search Engines Can’t See Your Website — and How to Fix It
  2. AI Search Crawlers: GPTBot, ClaudeBot, PerplexityBot & the Bots Reading Your Site
  3. How to Configure robots.txt for AI Crawlers (Without Going Invisible)
  4. Do AI Crawlers Render JavaScript? Rendering & Content Extractability
  5. How to Audit Your Site for AI Search Crawlability (Step-by-Step)
  6. Fixing AI Crawlability by Platform: Next.js, WordPress, Headless & CDN · you’re here
← The audit
What’s next
Becoming visible in AI search is a technical problem with a revenue outcome. If you'd rather have it diagnosed and fixed, here's how I can help.
Mihir Naik
About the author
Mihir Naik — Senior Product Manager (AI) at seoClarity, building Clarity ArcAI. Born in Surat, India; based in Toronto. In SEO since 2011. Available for consulting.
Read full bio →