Hybrid Search: BM25 Plus Dense Retrieval Plus Reranking

Three days after your RAG demo launched, a support engineer files a bug: users asking about "ERR_CONN_REFUSED" in your infrastructure docs are getting generic networking advice instead of the specific troubleshooting section that mentions that exact error string. You check the vector index — the embedding for "ERR_CONN_REFUSED" sits close to "connection failure" and "socket timeout" in the latent space, but the troubleshooting chunk that matters ranks 14th. Meanwhile, users asking "how do I rotate my API keys?" are getting the exact right chunk because that question has clear semantic overlap with the indexed content.

These are not bugs in your embedding model. They are the two fundamental retrieval failure modes: dense vectors fail on rare exact terms; keyword search fails on semantic variation. Every pure-vector RAG system in production has the first failure mode. Every pure-BM25 system has the second. The hybrid search stack exists because you need both indices, not one.

Why dense-only retrieval fails exactly where BM25 wins

The two retrieval methods fail in exactly opposite directions:

flowchart LR
    Q["User query"] --> BM25["BM25 index"]
    Q --> DENSE["Dense bi-encoder"]
    BM25 --> |"exact match wins"| EX["'ERR_CONN_REFUSED'\nproduct SKUs\nerror codes"]
    BM25 --> |"misses"| SYN["paraphrase\nsynonyms\ncross-lingual"]
    DENSE --> |"semantic match wins"| SYN2["'password reset'\n= 'forgot credentials'"]
    DENSE --> |"misses"| EX2["rare exact terms\nproper nouns\nlow-frequency codes"]
    style BM25 fill:#00e5ff,color:#0a0a0f
    style DENSE fill:#a855f7,color:#fff
    style EX fill:#15803d,color:#fff
    style SYN2 fill:#15803d,color:#fff

Embedding models are trained to put semantically similar text near each other in vector space. This makes them powerful for paraphrase — "password reset" and "forgot credentials" land close — but it means exact character sequences matter less than distributional context. A 6-character error code like "ERR_CONN_REFUSED" gets tokenized into subword pieces and embedded based on what typically surrounds such error codes in the training corpus, not based on exact match. If the indexed chunk contains that exact string and the query also contains it, a bi-encoder may still score another chunk higher if that other chunk is more generally about connection errors.

BM25 has the opposite property. It is a statistical formula over term frequency and inverse document frequency — a term that appears in the query and in a document gets scored proportionally to how rare that term is across the corpus. "ERR_CONN_REFUSED" appearing in both query and document gets a very high IDF score because it's rare. BM25 cannot handle synonymy ("rotate" vs. "refresh" for API keys), but for exact-match recall on rare, high-information terms, it is consistently better than any bi-encoder.

The complementarity is not hypothetical. On the BEIR benchmark suite — 18 heterogeneous retrieval datasets covering biomedical, legal, financial, and general web domains — hybrid systems consistently outperform both BM25-only and dense-only baselines. The gap is especially large on datasets with technical jargon, product codes, and named entities.

BM25: what it actually computes

BM25 is not just counting word occurrences. The formula applies a saturation function to term frequency (so 10 occurrences of a query term in a document doesn't score 10× better than 1 occurrence — relevance saturates), and normalizes for document length (so a 5,000-word document containing a query term once doesn't automatically beat a 200-word document that mentions it three times).

The two parameters are k1 (controls term frequency saturation, typically 1.2–2.0) and b (document length normalization, typically 0.75). These defaults work well across most corpora. The practical implication: BM25 is a stateless index built from the same chunks you embed. Running both requires only double the index storage — the query overhead is minimal because BM25 runs on an inverted index, not the embedding space.

In Elasticsearch, Qdrant (with sparse vectors enabled), Weaviate, and OpenSearch, BM25 is the default text search. You already have it if you're using these systems; you may just not be querying it alongside your vector index.

Dense retrieval: where bi-encoders break down

The canonical bi-encoder setup encodes the query and each document with the same model, then retrieves the nearest documents by cosine similarity using an HNSW index. The critical point is that encoding is independent: the model sees the query alone and the document alone, never together. This is what makes ANN search feasible over millions of documents — you precompute all document embeddings offline and search at query time.

Independence is also the weakness. Cross-attention between query and document tokens — the mechanism that makes reading comprehension models good at relevance — is exactly what bi-encoders lack by construction. They have to compress the entire meaning of a passage into a fixed-size vector, which loses fine-grained relevance signals.

The practical symptom in RAG: bi-encoders consistently surface chunks that are topically related to the query but not specifically answering it. If your corpus contains 50 chunks about authentication and the user asks a specific question about OAuth token expiry, the bi-encoder will retrieve 5 authentication chunks, but not necessarily the one that answers the question. The reranker fixes this — but we're getting ahead of ourselves.

SPLADE: the neural upgrade for BM25

SPLADE (SParse Lexical AnD Expansion) is a learned sparse retrieval model. It runs a BERT-style encoder and applies a RELU + log activation to the output logits, producing a sparse vector over the full vocabulary where each dimension represents a term and its weight. Unlike BM25, which can only score terms that appear literally in the query and document, SPLADE implicitly expands both — a query about "password rotation" might activate the "credentials", "auth token", and "API key" dimensions even without those words appearing.

On BEIR, SPLADE consistently outperforms BM25 across most domains, with nDCG@10 improvements of 5–15 percentage points depending on the dataset. The cost: SPLADE requires a neural forward pass per document at indexing time and per query at search time, adding roughly 100–300ms of inference latency. For an interactive RAG pipeline with a sub-500ms retrieval budget, that's borderline. For a batch pipeline or an application where generation latency dominates, it's straightforwardly worth it.

Whether to use BM25 or SPLADE as your sparse component is a decision to make per corpus. Benchmark both on your eval set (see RAG Evaluation with Ragas for how) before committing.

Reciprocal Rank Fusion: the right way to merge ranked lists

The naive approach to fusing BM25 and dense results is to normalize each list's scores to [0,1] and take a weighted sum. This breaks because BM25 scores are not comparable to cosine similarity scores — a BM25 score of 12.3 and a cosine similarity of 0.87 have no common unit. Normalization helps but is sensitive to outliers and depends on the score distribution of each query.

Reciprocal Rank Fusion sidesteps score comparison entirely. Given ranked lists L₁ and L₂:

RRF_score(doc) = Σ  1 / (k + rank(doc, Lᵢ))
                 i

where k = 60 (standard constant, downweights tail ranks)

A document ranked 1st in both lists gets 1/(60+1) + 1/(60+1) ≈ 0.033. A document ranked 1st in one list but absent from the other gets only 0.016. A document ranked 20th in both lists gets about 0.025 — still competitive, because RRF's log-scale-like decay is gentle.

The k=60 constant was chosen empirically across information retrieval benchmarks and remains the standard default. You can introduce a weight multiplier per list to shift the balance between sparse and dense:

def rrf_score(rankings: list[dict[str, int]], weights: list[float], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking, weight in zip(rankings, weights):
        for doc_id, rank in ranking.items():
            scores[doc_id] = scores.get(doc_id, 0.0) + weight * (1.0 / (k + rank))
    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

# 60% dense, 40% BM25 — a reasonable starting point
fused = rrf_score(
    rankings=[dense_ranks, bm25_ranks],
    weights=[0.6, 0.4],
)

The weight ratio is a tunable parameter. 60/40 dense/BM25 is the most commonly cited starting point, but it's corpus-dependent. Measure RAGAS Context Recall across several weight ratios on a representative sample of queries before locking it in.

{ "type": "hybrid-search", "alpha": 0.6, "title": "BM25 + Dense fusion with RRF — drag the weight slider" }

The cross-encoder reranker: where precision comes from

After RRF fusion you have roughly 20–40 candidates ranked by a score that's a combination of term-matching and semantic similarity. These candidates have high recall — the correct chunk is almost certainly in this set — but their ordering is still imprecise. The cross-encoder reranker fixes the ordering.

A cross-encoder takes the full (query, passage) pair as input and runs a BERT-style model over the concatenated text. This allows attention between every query token and every passage token — exactly the cross-attention that bi-encoders sacrifice for speed. The output is a single relevance score, much more accurate than the bi-encoder's independent encoding.

flowchart TD
    subgraph BI["Bi-encoder (recall stage)"]
        Q1["Query"] --> EQ["Encode query\n→ vector"]
        D1["Doc 1"] --> ED1["Encode doc\n→ vector (offline)"]
        D2["Doc N..."] --> EDN["Encode doc\n→ vector (offline)"]
        EQ --> CS["Cosine similarity\n(ANN search)"]
        ED1 --> CS
        EDN --> CS
    end
    subgraph CE["Cross-encoder (rerank stage)"]
        QD["Query + Passage\n(concatenated)"] --> BERT["BERT encoder\nfull cross-attention"]
        BERT --> SC["Relevance score"]
    end
    CS --> |"top 20 candidates"| CE
    style BI fill:#0e7490,color:#fff
    style CE fill:#a855f7,color:#fff

The tradeoff is that cross-encoders cannot index a large corpus: scoring 1 million documents at query time would require 1 million forward passes. They're only viable as rerankers over a small candidate set.

The standard pattern:

from cohere import Client

co = Client(api_key="...")

def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[dict]:
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=candidates,
        top_n=top_n,
    )
    return [
        {"index": r.index, "score": r.relevance_score, "text": candidates[r.index]}
        for r in response.results
    ]

# Retrieve 20 candidates from BM25 + dense + RRF
candidates = retrieve_hybrid(query, top_k=20)
# Rerank to top 5 for LLM context
top_5 = rerank(query, [c["text"] for c in candidates], top_n=5)

If you prefer to run locally, BAAI/bge-reranker-v2-m3 is the strongest open-weight cross-encoder as of mid-2026. FlashRank wraps several reranker models with optimized inference and is the fastest local option.

Milliseconds are only half the reranker's price tag. Hosted rerank APIs bill per search, and at production volume that line item sneaks up on you:

Reranker cost, hosted vs. self-hosted (illustrative, as of mid-2026):

Cohere Rerank: ~$2.00 per 1,000 searches
  10K queries/day  → ~300K searches/month  → ~$600/month
  50K queries/day  → ~1.5M searches/month  → ~$3,000/month

Self-hosted bge-reranker-v2-m3 on one 24GB GPU (L4-class, ~$0.70/hr):
  ~$500/month flat, handles 50K queries/day with headroom
  — but you own model serving, scaling, and the 3am pages

Crossover: somewhere around 10–20K queries/day, self-hosting starts
winning on dollars. Below that, the API is cheaper than the GPU it
would replace — and often costs more per month than your embeddings.

At small scale the hosted API is the obvious call. At 50K+ queries/day, the reranker can be your single largest retrieval line item, and a quantized local BGE model on one GPU pays for itself in weeks.

HyperRAG (2025) introduced KV-cache reuse for cross-encoders: because the passage representations can be cached across queries that share context, the effective reranking latency drops from 80–200ms to roughly 30–60ms. This makes the reranker viable in interactive pipelines where the 80ms tax was previously unacceptable.

Contextual Retrieval: making BM25 smarter without changing BM25

Anthropic's Contextual Retrieval (September 2024) is a complementary technique that improves BM25 (and dense) retrieval by enriching chunks before indexing, rather than by changing the retrieval mechanism.

The idea: before indexing a chunk, prepend a 50–100 token LLM-generated summary that situates the chunk within the broader document — "This section describes the OAuth 2.0 token rotation procedure in the context of the API authentication guide." The enriched chunk goes into both the BM25 inverted index and the embedding model. A query about "how to refresh my OAuth token" now finds relevant BM25 term overlap with "OAuth", "token", and "rotation" even if the raw chunk only says "call the /refresh endpoint."

The result from Anthropic's evaluation: contextual embeddings alone reduced retrieval failures by 35%. Contextual embeddings plus contextual BM25 — hybrid search over the enriched chunks — brought that to 49%. Adding a cross-encoder reranker on top of that stack: 67% fewer failures. Note that hybrid search is already inside the 49% and 67% figures; there is no extra hybrid bonus to stack on top. The technique requires one LLM call per chunk at indexing time, but with prompt caching enabled, costs roughly $1.02 per million document tokens — competitive with embedding costs.

The catch: Contextual Retrieval without prompt caching is significantly more expensive. Enabling prefix caching on the document context sent to the LLM is what makes the economics work. See Prompt Caching: Cutting Costs 50–90% for how to implement this.

Full pipeline implementation

Here is the production pattern in full, using Qdrant's hybrid search API (which natively supports BM25 sparse vectors alongside dense vectors) and Cohere Rerank:

from qdrant_client import QdrantClient, models
from cohere import Client as CohereClient
from fastembed import SparseTextEmbedding, TextEmbedding

qdrant = QdrantClient(url="http://localhost:6333")
cohere = CohereClient(api_key="...")

# Load models
dense_model = TextEmbedding("BAAI/bge-base-en-v1.5")
sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")  # or BM25


def hybrid_retrieve(query: str, collection: str, top_k: int = 20) -> list[dict]:
    dense_vec = list(dense_model.embed([query]))[0].tolist()
    sparse_vec = list(sparse_model.embed([query]))[0]

    results = qdrant.query_points(
        collection_name=collection,
        prefetch=[
            models.Prefetch(
                query=dense_vec,
                using="dense",
                limit=top_k,
            ),
            models.Prefetch(
                query=models.SparseVector(
                    indices=sparse_vec.indices.tolist(),
                    values=sparse_vec.values.tolist(),
                ),
                using="sparse",
                limit=top_k,
            ),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=top_k,
        with_payload=True,
    )
    return [
        {"id": r.id, "text": r.payload["text"], "score": r.score}
        for r in results.points
    ]


def retrieve_and_rerank(query: str, collection: str, final_k: int = 5) -> list[str]:
    candidates = hybrid_retrieve(query, collection, top_k=20)
    texts = [c["text"] for c in candidates]

    reranked = cohere.rerank(
        model="rerank-v3.5",
        query=query,
        documents=texts,
        top_n=final_k,
    )
    return [texts[r.index] for r in reranked.results]

Qdrant handles RRF fusion server-side. Weaviate and Vespa have equivalent APIs. If you're on Elasticsearch or OpenSearch, the hybrid query endpoint also supports reciprocal rank fusion — check their linear_combination vs rrf parameter in the hybrid query type.

What breaks

Wrong BM25 / dense weight ratio for your corpus

The 60/40 dense/BM25 split is a starting point, not a law. A corpus of internal engineering runbooks written in precise technical language may benefit from a heavier BM25 weight. A corpus of informal customer support tickets, full of typos and non-standard terminology, may benefit from a heavier dense weight. Tune this on a representative eval set with ground-truth relevance labels. Running 5–7 weight combinations takes under an hour with the right evaluation harness.

Reranker latency blowing the SLA

Before HyperRAG-style KV-cache reuse, cross-encoder reranking of 20 candidates at 80–200ms was a real latency problem for sub-500ms retrieval targets. The options: use FlashRank with a quantized model (~30ms), apply KV-cache reuse if your framework supports it, or reduce the candidate set to 10 with a small precision penalty. Don't skip the reranker to hit an SLA — the precision difference is large enough to matter in user-visible answer quality. Instead, parallelize retrieval and start streaming the LLM response before reranking completes if your architecture allows it.

BM25 tokenization mismatch

BM25 in different systems tokenizes differently. Elasticsearch uses its own stemming and normalization. Qdrant's sparse vector approach requires you to choose a tokenizer when building the sparse index. If your chunk indexing tokenizes "OAuth" as one token and "oauth" as another, a query with different capitalization will miss the match. Use a consistent, lowercasing tokenizer across your indexing and query pipelines. Test with a few synthetic queries that you know should match before going to production.

Reranker hallucination on long passages

Cross-encoder models have their own context window limits — most are trained on passages of 512 tokens or less. If your chunks are longer (512–1,024 tokens), the reranker may truncate them, scoring only the first 512 tokens. This creates a systematic bias: the beginning of a long chunk gets scored, the end does not. Either keep chunks within 512 tokens (a good practice anyway — see Chunking Strategies That Actually Matter) or use a reranker fine-tuned for longer contexts.

Lost-in-the-middle after retrieval

Retrieval precision matters, but so does how you order the retrieved chunks in the LLM prompt. Research consistently shows that LLMs pay the most attention to the start and end of their context, with accuracy degrading 30%+ for facts placed in the middle. After reranking, put the highest-scoring chunk first and second-highest last — don't just concatenate chunks in reranker output order. This has zero cost and measurable accuracy impact.

{ "type": "lost-middle", "title": "Chunk position vs. LLM attention — drag the relevant chunk" }

Skipping evaluation of the retrieval stage

The most common production mistake in RAG is treating retrieval as a solved problem and blaming the LLM when answers are wrong. Low RAGAS Faithfulness scores almost always trace to retrieval, not generation — the LLM is faithful to whatever context you give it, including the wrong context. Measure Context Precision (are retrieved chunks relevant?) and Context Recall (did you retrieve the chunks that matter?) separately from generation quality. Fix retrieval first. See RAG Evaluation, Failure Modes for the full evaluation pipeline.

When to use what: the decision in practice

Not every RAG system needs all three stages. Here's when each layer justifies its cost:

BM25 only works if your queries are primarily keyword-based, your corpus is technical documentation with precise terminology, and you've measured that adding dense retrieval doesn't move your eval metrics. Rare — most production systems see a clear benefit from adding dense retrieval.

Dense-only is reasonable for a prototype or a corpus with conversational, non-technical content where exact term overlap is unlikely to be the primary failure mode. Once you're measuring and see exact-term recall failures, add BM25.

BM25 + dense + RRF (no reranker) is a good first production configuration. It's fast, improves recall reliably, and requires no additional API call. Start here before paying for a reranker.

BM25 + dense + RRF + cross-encoder reranker is the consensus production stack as of 2025–2026. The reranker's precision improvement is consistently large enough to justify the 80–200ms cost. Budget the dollars too: per the cost block above, a hosted reranker at 50K queries/day runs around $3,000/month — the point where a self-hosted quantized BGE-reranker on one GPU becomes the cheaper option. Use FlashRank or a quantized BGE-reranker for latency-sensitive deployments.

SPLADE + dense + RRF + cross-encoder is the accuracy-maximizing configuration when you have the latency budget. SPLADE's implicit query expansion helps on corpora where synonym recall is a significant failure mode. Benchmark on your data before committing to the additional indexing and query-time compute.

Configuration	Latency	Precision	When to reach for it
Dense only	10–30ms	Baseline	Prototyping, conversational corpora
BM25 only	5–20ms	Baseline for exact-term queries	Keyword-heavy domains, legacy compatibility
BM25 + Dense + RRF	20–40ms	Meaningfully better than either alone	First production configuration
+ Cross-encoder reranker	80–240ms	Much better precision	Standard production stack
+ SPLADE instead of BM25	180–500ms	Best recall on synonym-heavy queries	Accuracy-critical, higher latency budget
+ Contextual Retrieval	+indexing cost	Best precision overall	When you have prompt caching and want maximum quality

One more thing to watch: Query Transformation operates on the query side and is complementary to hybrid search. A rewritten or HyDE-expanded query goes through the same BM25 + dense + RRF + reranker pipeline and benefits at every stage. These are not competing techniques — stack them if your eval metrics justify both costs.

The retrieval stack described here — hybrid search with RRF fusion and a cross-encoder reranker — is not experimental. It's what Qdrant, Weaviate, Vespa, and OpenSearch all now support natively, and it's the configuration you should have running in production before considering anything fancier. Graph retrieval and agentic loops (covered in GraphRAG and Structured Retrieval and Agentic RAG) build on top of a retrieval layer that already works. Get this one right first.