What Is a Reranker? Why RAG Pipelines Need One

A reranker is a second-stage model in a search or retrieval-augmented generation pipeline that re-scores a shortlist of candidate results for relevance to the query, using a more accurate — but slower — comparison than the initial retrieval step. It sits between “get some plausible documents fast” and “hand the best ones to the model,” fixing precision problems that vector search alone tends to leave behind.

Why retrieval alone isn’t enough

Most RAG systems retrieve candidates using vector similarity search — embedding the query and the documents, then finding nearest neighbors with an index like HNSW. This is fast and scales to millions of documents, but it has a structural weakness: it scores each document by comparing its embedding to the query’s embedding independently, with no way to directly weigh the query against the document’s full text at query time.

That approximation is good enough to narrow millions of documents down to a few dozen plausible candidates. It’s often not good enough to correctly rank those candidates — a document that’s topically close but doesn’t actually answer the question can outscore one that does, purely because of how it happened to land in embedding space.

What a reranker actually does

A reranker takes the query and a small set of retrieved candidates — typically the top 20 to 100 from the retrieval step — and scores each query-document pair jointly, usually with a cross-encoder model. Instead of comparing two independently computed vectors, a cross-encoder feeds the query and the document text into the same model at once, so it can attend to how specific words and phrases in the document relate to specific words in the query.

That joint scoring is far more accurate, but also far more expensive: it requires a full model forward pass per query-document pair, rather than a single vector comparison against a pre-built index. That’s why reranking is a second stage applied to a shortlist, not a replacement for retrieval — running a cross-encoder over an entire corpus for every query wouldn’t scale.

A typical pipeline looks like:

1. Retrieve — vector search (or hybrid vector + keyword search) pulls the top 50-100 candidates from the full corpus.
2. Rerank — a cross-encoder scores each candidate against the query and re-sorts them.
3. Select — the top 3-10 reranked results are passed into the model’s context window, shaped by whatever chunking strategy built the corpus in the first place.

Where the cost shows up

Reranking adds latency and compute that pure vector search doesn’t have — one model call per candidate document, on top of the retrieval step and the eventual generation call. For a shortlist of 50 candidates, that’s 50 small inference calls before the main model even sees a prompt. Teams tracking LLM costs should treat reranking as its own line item, separate from generation; our token cost calculator is built around generation pricing, so reranker cost needs to be estimated separately based on the reranking model’s own pricing and the candidate-list size.

The usual way to control that cost is tuning how many candidates get reranked. Reranking the top 100 catches more relevant documents that retrieval ranked poorly, but costs more than reranking the top 20. Most production systems settle somewhere in between after measuring the tradeoff against their own corpus.

Rerankers vs the retriever itself

It helps to be precise about what each stage is actually doing, since “search relevance” gets used loosely for both.

	Retriever (vector search)	Reranker (cross-encoder)
Scores	Query and document independently	Query and document jointly
Speed	Fast — index lookup	Slow — one model call per pair
Scale	Millions of documents	Tens to low hundreds of candidates
Accuracy	Approximate, sometimes noisy	High precision on the shortlist it sees
Role in pipeline	Narrows the search space	Refines the final ranking

Neither stage replaces the other. Retrieval without reranking is fast but imprecise at the margins; reranking without retrieval doesn’t scale at all.

When reranking matters most

Reranking earns its cost most clearly when precision in the top few results directly affects answer quality — which is most RAG use cases, since only a handful of chunks make it into the model’s context. It matters less for exploratory or recall-oriented search, where a user is willing to scan a longer list and precision in the top 3 is less critical.

It also compounds well with hybrid retrieval, where the initial candidate set already combines vector similarity with keyword or BM25 matches. The reranker doesn’t care how a candidate was retrieved — it only judges the query-document pair — so mixing retrieval strategies and reranking the combined shortlist tends to outperform either strategy alone.

The takeaway

A reranker is a precision fix for the recall-oriented approximation that vector search makes: it re-scores a small shortlist of candidates with a slower, joint query-document model, then hands the reordered top results to the generation step. It adds latency and per-call cost, so it’s applied to dozens of candidates, not the whole corpus — but for RAG pipelines where only a few chunks reach the context window, that added precision is often the difference between a relevant answer and a plausible-sounding wrong one.