GuidePublished: June 30, 2026 · Last updated: June 30, 2026 · ~6 min read

Mihir Naik — Senior PM (AI) at seoClarity

Do AI Crawlers Render JavaScript? Rendering & Content Extractability

You can pass every access check and still be invisible—because the crawler fetched your page and found an empty shell. The major AI crawlers download HTML but don’t run JavaScript; if your content is assembled in the browser, it isn’t there when the bot looks. This part covers the evidence, why it happens, and how to make your content present and extractable.

Executive summary

Reaching your page (access) and reading your page (rendering) are separate problems. The major AI crawlers fetch raw HTML and do not execute JavaScript, so client-side-rendered content is effectively invisible to them—even when the page looks perfect in a browser.

If your content only appears after JavaScript runs, assume AI crawlers can’t see it.
Server-side rendering (SSR) or static generation (SSG) puts your content in the initial HTML, where crawlers actually look.
Rendering is necessary but not sufficient—content also has to be extractable: semantic, in the main document, not hidden behind interaction.

On this page

Do AI crawlers render JavaScript?

As a working assumption: no. A network-scale analysis by Vercel and MERJ (Dec 2024) found no evidence of JavaScript execution across hundreds of millions of GPTBot requests. The crawlers often download JS files—GPTBot fetched them in ~11.5% of requests, ClaudeBot in ~23.8%—but they don’t run them. The same non-rendering behavior held for ClaudeBot, PerplexityBot, Meta’s crawler, and ByteSpider. For scale, that study saw GPTBot generate ~569M requests and ClaudeBot ~370M in a single month.

Two exceptions: Google’s Gemini reuses Googlebot’s rendering pipeline, and Applebot renders pages—so ‘AI crawlers don’t render JS’ is true of OpenAI/Anthropic/Perplexity/Meta today, not a permanent law. Treat it as a present-state finding and re-verify.

The important nuance is fetch versus execute. Non-rendering crawlers receive your server’s initial HTML response and parse it as text—but never run the bundle, so any DOM your JavaScript would build never materializes for them. (Googlebot, by contrast, crawls, then renders with headless Chromium, then indexes—but even it queues rendering.)

Design for the lowest common denominator: if your important content isn’t in the raw HTML response, treat it as invisible to AI.

Sources & further reading

The rise of the AI crawler — Vercel + MERJ (Dec 2024) — Primary empirical evidence that major AI crawlers fetch but don’t execute JS.
Understand JavaScript SEO basics — Google Search Central — Google’s crawl → render → index pipeline (the exception AI crawlers aren’t).

How client-side rendering hides your content

In a client-side-rendered app (a typical single-page React/Vue/Angular build), the server sends a near-empty HTML shell plus a JavaScript bundle. The browser runs that bundle to build the page. A crawler that doesn’t run JavaScript receives only the shell.

What a non-rendering crawler sees from a CSR app

<!doctype html>
<html>
  <head><title>Loading…</title></head>
  <body>
    <div id="root"></div>
    <script src="/static/bundle.js"></script>
  </body>
</html>

Every headline, paragraph, and proof point lives inside that bundle’s runtime output—none of it is in the markup the bot reads. The practical casualties are exactly the pages that should win AI citations: pricing tables, dynamically rendered FAQs and specs, and API-loaded comparison data. (A related trap: Google deprecated the old AJAX-crawling `#!` scheme in 2015, so fragment-loaded views aren’t a workaround.)

For leadership

If your site is a single-page app, this is likely your single biggest AI-visibility risk—and it won’t show up in a browser QA pass, because your browser runs the JavaScript the crawler won’t.

Sources & further reading

Client-side Rendering (CSR) — Next.js docs — The shell + JS pattern defined by the framework.
Fix search-related JavaScript problems — Google — SPA pitfalls and verifying content is in the HTML.

The fix: render on the server

Put your content in the HTML before it leaves the server. Three well-supported approaches, all of which produce the same crawler-visible result—your real content in the initial HTML response:

Server-side rendering (SSR): the server renders the page’s HTML per request. Good for dynamic/personalized pages.
Static site generation (SSG): pages are pre-rendered to HTML at build time and served as static files—ideal for content like guides, docs, and marketing pages, and CDN-cacheable.
Incremental static regeneration (ISR): static generation per page without rebuilding the whole site, so it scales to large content sets and refreshes on a schedule or on demand.

Hydration can still add interactivity afterward—what matters is that the content doesn’t depend on it. Notably, Google now says dynamic rendering is no longer a recommended long-term workaround; prefer SSR, static rendering, or hydration.

You don’t have to rewrite everything. Often the highest-value pages (key product, solution, and resource pages) can be moved to SSR/SSG first, capturing most of the visibility upside.

Sources & further reading

Incremental Static Regeneration (ISR) — Next.js docs — Static output that refreshes without full rebuilds.
Dynamic rendering as a workaround — Google — Google steers away from dynamic rendering toward SSR/SSG.

Beyond rendering: making content extractable

Rendering gets your content into the HTML. Extractability decides whether a model can isolate, attribute, and reuse it cleanly. Even server-rendered pages can bury their value.

What helps a machine extract you accurately:

Semantic HTML—real headings, paragraphs, lists, and clear landmarks (a `<main>` for primary content, `<article>`/`<section>` with headings) instead of a sea of unlabeled divs.
Your primary content in the served HTML, not collapsed inside tabs/accordions that only populate on click, or loaded by infinite scroll—these are the same non-rendering problem as CSR.
Important claims as text, not baked into images or screenshots a text crawler can’t read.
A clear content hierarchy so the answer to a question sits under a heading that matches the question. Structured data (Schema.org) adds a machine-readable layer on top.

Tabs and ‘load more’ are great UX and quiet visibility killers. If content matters for AI answers, make sure it exists in the served HTML, not just after an interaction.

What about llms.txt? The proposed /llms.txt file (2024) offers a curated, LLM-friendly map of your site—but no major AI provider has committed to using it, real-world hit rates are near zero, and Google has called it ‘purely speculative.’ It’s low-cost and low-risk, but it is not a substitute for putting content in server-rendered HTML.

Sources & further reading

Semantic HTML & the <main> element — MDN — Landmarks and document outline for machine extraction.
Google says llms.txt is ‘purely speculative for now’ — Search Engine Journal — Why llms.txt isn’t a crawlability fix today.
Fix lazy-loaded content — Google — Don’t gate content behind scroll/interaction.

Test what the crawler actually sees

Stop trusting the rendered browser view. Inspect the raw HTML the way a non-rendering crawler does. The gap between View Source (served HTML) and Inspect/DevTools (post-JavaScript DOM) is exactly the content AI crawlers miss.

See the raw, pre-JavaScript HTML

# Is your key content in the raw HTML?
curl -s https://example.com/your-page | grep -i "your key phrase"

# Missing here but present in the browser = client-side only,
# and likely invisible to AI crawlers.

Two more quick checks:

Right-click → View Source (not Inspect). Source shows the served HTML; Inspect shows the post-JavaScript DOM.
Disable JavaScript in your browser and reload. What remains is roughly what a non-rendering crawler gets. (Note: Google’s URL Inspection tool shows the rendered view—useful, but it’s Googlebot’s render, not GPTBot’s raw fetch.)

Sources & further reading

Fix search-related JavaScript problems — Google — View Source vs rendered HTML; testing methods.

Part 4 of 6

A part of Why AI Search Engines Can’t See Your Website — and How to Fix It.

Why AI Search Engines Can’t See Your Website — and How to Fix It
AI Search Crawlers: GPTBot, ClaudeBot, PerplexityBot & the Bots Reading Your Site
How to Configure robots.txt for AI Crawlers (Without Going Invisible)
Do AI Crawlers Render JavaScript? Rendering & Content Extractability · you’re here
How to Audit Your Site for AI Search Crawlability (Step-by-Step)
Fixing AI Crawlability by Platform: Next.js, WordPress, Headless & CDN

← robots.txt & CDN The audit →

What’s next

Becoming visible in AI search is a technical problem with a revenue outcome. If you'd rather have it diagnosed and fixed, here's how I can help.

About the author

Mihir Naik — Senior Product Manager (AI) at seoClarity, building Clarity ArcAI. Born in Surat, India; based in Toronto. In SEO since 2011. Available for consulting.

Read full bio →