Hybrid Search and Reranking: Better RAG Retrieval

Most RAG systems ship with a single retriever: a vector database that turns the query into an embedding and finds the nearest chunks by cosine similarity. It works fine for paraphrased, conceptual questions, and it falls apart the moment a user types an exact product code, an error message, or a rare acronym that never appeared anywhere near similar text during embedding training.

This post walks through the two changes that consistently move retrieval quality the most in production RAG: fusing lexical search with vector search using reciprocal rank fusion, and adding a cross-encoder reranking pass before the final context ever reaches the language model.

Why vector search alone is not enough

Dense embeddings are trained to capture meaning, which is exactly why they miss exact matches. A vector model may place the string of a specific part number close to unrelated part numbers, because it has learned that identifiers in general are similar to each other, not that this identifier is unique.

• Product codes, SKUs, and error codes lose their exactness once compressed into a fixed-size vector.
• Rare technical acronyms and proper nouns often get embedded near generic, unrelated terms.
• Negations and numeric constraints, like a specific version number or a currency amount, tend to blur together in embedding space.

Lexical search, in the form of BM25, handles exactly these cases because it scores documents on literal term overlap, weighted by how rare each term is across the whole collection. Running both retrievers side by side and combining their outputs recovers the strengths of each without inheriting the weaknesses of either.

Reciprocal rank fusion: combining two ranked lists

Reciprocal rank fusion, or RRF, is the standard answer to the score-incompatibility problem. Instead of combining raw relevance scores, RRF only looks at each document's rank position in every list it appears in. A document ranked first in the BM25 list and third in the vector list gets a combined score built purely from those two rank positions.

The formula sums one divided by a constant plus the rank, across every list a document appears in. The constant, usually set to sixty, dampens the influence of documents that only barely made it into a list. Documents that show up near the top of both lists rise to the top of the fused ranking, which is exactly the behavior you want from a retriever feeding an LLM.

Implementing reciprocal rank fusion

The implementation is short enough to write by hand even if your vector database or search engine already offers a built-in hybrid query. Run both retrievers against the same query, collect their ranked document lists, then merge with the RRF formula below.

function reciprocalRankFusion(bm25Ranked, vectorRanked, k = 60) {
  const scores = new Map();

  const addRanks = (rankedList) => {
    rankedList.forEach((doc, index) => {
      const rank = index + 1;
      const contribution = 1 / (k + rank);
      scores.set(doc.id, (scores.get(doc.id) || 0) + contribution);
    });
  };

  addRanks(bm25Ranked);
  addRanks(vectorRanked);

  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .map(([id, score]) => ({ id, score }));
}

Notice that the fusion function never looks at the raw BM25 score or the raw cosine similarity, only at list position. That is the entire point: rank-based fusion sidesteps the need to normalize two incomparable scoring systems onto the same scale.

Adding a cross-encoder reranker

Fusion gets you a better candidate set, but it still ranks documents using two relatively shallow signals: term overlap and vector distance. A cross-encoder reranker closes that gap by jointly encoding the query and each candidate passage through a single transformer pass, producing one relevance score per pair instead of comparing two independently computed representations.

1. Retrieve a wide candidate set, commonly fifty to one hundred chunks, using the fused hybrid search from the previous step.
2. Pass each query-passage pair through the cross-encoder model, one pair at a time.
3. Sort candidates by the reranker's relevance score, not by the original fusion score.
4. Keep only the top eight to fifteen reranked passages and hand those to the generation model.

Because the reranker attends over the query and passage together, it can catch relationships that neither BM25 nor a vector similarity score can see, such as a passage that mentions all the right keywords but answers a completely different question. That precision comes at a real latency cost, which is why reranking always runs on a narrowed candidate set, never on the full corpus.

Measuring retrieval quality before you touch generation

Teams frequently jump straight to tweaking prompts when a RAG answer is wrong, without first checking whether the retriever ever surfaced the right passage. Separating retrieval evaluation from generation evaluation is the single most useful debugging habit in RAG work.

• Build a small labeled set of queries with the correct source passage marked for each one, even fifty examples is enough to start.
• Measure recall at k, the fraction of queries where the correct passage appears anywhere in the top k retrieved results, before and after adding hybrid search and reranking.
• Track normalized discounted cumulative gain, or NDCG, which additionally rewards the correct passage appearing near the top of the list rather than merely somewhere in it.

If recall at k is low, no amount of reranking or prompt engineering fixes the problem, because the right passage was never in the candidate set to begin with. If recall is high but NDCG is low, that is precisely the situation a reranker solves.

Putting it together in a production pipeline

A pragmatic hybrid search and reranking pipeline looks like this: run BM25 and vector search in parallel against the same chunk store, fuse the two ranked lists with reciprocal rank fusion, take the top candidates from the fused list, rerank those with a cross-encoder, and pass only the final handful of passages to the generation model.

Each stage narrows the candidate set while raising precision, which keeps the expensive cross-encoder pass fast and keeps the context window handed to the LLM small and relevant rather than padded with marginal matches. This layered design also makes the pipeline easier to debug in isolation, since a regression in final answer quality can usually be traced back to one specific stage rather than to the system as a whole.