How AI Retrieves and Ranks Passages (Not Pages)

When an engine retrieves, it isn’t crawling the open web live and reading your latest page in real time. It searches an index, a pre-built, searchable copy of the web, and different engines use different indexes: their own, or a partner’s. Perplexity and ChatGPT lean on large web indexes; Google’s AI surfaces use Google’s index; Claude relies on a third-party search provider. The next section breaks down whose index each engine actually uses. The practical effect is that the pool of candidate pages is different on every engine, and it is only ever as fresh and as complete as that index.

Two things follow. First, if your page isn’t in the index, it cannot be retrieved, no matter how good it is. That’s the whole subject of the companion series on why AI search engines can’t see your website. Second, being in the index is necessary but not sufficient: the index might hold millions of eligible pages, and only a few are pulled for any single query.

Because retrieval can only reach what’s in the index, it helps to know whose index each assistant actually uses. Not every arrangement is officially confirmed, and they shift over time, but the broad picture is well-evidenced, and it changes how you should think about ‘getting indexed.’

So the index layer clusters into a few camps: Bing (ChatGPT, Copilot, DuckDuckGo), Google (Gemini and AI Overviews), and Brave (Claude), plus each engine’s own growing crawl layer on top. Being present in Bing’s and Google’s indexes covers most of the surface, and Claude adds Brave.

Yes, and more than most people expect. The ‘index’ isn’t one uniform pool the engine draws from evenly. The candidate set it pulls depends heavily on what kind of question you asked, because different industries have different authoritative source ecosystems. Analyses of AI citations consistently find that the mix of dominant sources shifts sharply from one category to the next.

A handful of sources recur almost everywhere: Wikipedia, YouTube, Reddit, and a few large platforms show up across nearly every vertical. Underneath that shared layer, though, the pattern is specific to your market:

Which engine you’re looking at changes the mix again. Reports find Perplexity leans heavily on Reddit and review platforms, ChatGPT leans on encyclopedic sources and third-party directories and brand mentions, and Google’s AI surfaces lean on YouTube and Google’s own ecosystem, Business Profiles and the Knowledge Graph, for many queries. One analysis found only around 11% of cited domains overlapped between ChatGPT and Perplexity. And it moves: the most-cited domains turn over quickly, so treat any list as a snapshot, not a fixed map.

Before anything is indexed for semantic search, long pages are split into chunks, passages of roughly a few hundred words. This isn’t a stylistic choice; it’s a hard constraint. The models that encode text into searchable form and the models that generate answers both have token limits and can’t process an entire long document in one pass, so documents are broken into self-contained pieces that each carry one coherent idea.

This passage-level view isn’t unique to AI chat. Google moved the same direction years ago with passage ranking, announced in 2020: it started ranking a specific, highly relevant section of a page even when the rest of the page was about something else, affecting roughly 7% of queries. The lesson generalizes. Machines increasingly evaluate the passage, not the page.

The practical implication is about structure. If your best answer only makes sense after three preceding paragraphs of setup, the chunk that gets indexed may not stand on its own, and a passage that can’t stand on its own retrieves poorly. Sections that state their point directly, under a clear heading, survive chunking intact.

The engine matches on meaning. Both your passage and the query are turned into embeddings, lists of numbers that act as a fingerprint of meaning, so that two pieces of text that mean the same thing land close together even if they share no words. Finding relevant passages then becomes a geometry problem: retrieve the passages whose vectors sit nearest the query’s vector, measured by cosine similarity or dot product. This approach, dense passage retrieval, was shown to beat classic keyword search (BM25) substantially on open-domain question answering.

That’s why keyword stuffing does nothing here: matching is semantic, so a passage that genuinely explains a concept will be found under many different phrasings of the question. In practice most production systems run hybrid search, combining this semantic matching with old-fashioned keyword matching, so exact terms and names still matter, they just aren’t the whole game.

First-stage retrieval is built for speed, not precision. It casts a wide net and returns a large candidate set, often hundreds of passages, because at that scale each passage and the query are encoded separately and compared with fast math. That’s great for recall, but the ordering is rough. So a second stage re-reads the shortlist much more carefully.

That second stage is a reranker, typically a cross-encoder: instead of encoding the query and passage separately, it reads them together and scores how well that specific passage answers that specific query. It’s slower, so it’s only run on the shortlist, but it’s far more accurate. The technique that established this re-scored the top candidates from a fast first stage and improved ranking quality dramatically. Only the top few passages, often three to five, survive to be handed to the model that writes the answer.

Now the frustrating pattern resolves. ‘We have a great page and still don’t get cited’ almost always traces to one specific stage of this funnel, and each has a different fix.