RAG in One Pass: Build, Measure, Improve

The Tuesday your docs team migrated to a new CMS, your RAG bot's retrieval hit-rate dropped from 84% to 31%. You didn't know, because you weren't measuring hit-rate. For eleven days the bot kept answering fluently from whatever chunks it could still find — a customer asking about enterprise SSO got setup instructions for a product tier you discontinued last year. Nothing crashed. Nothing logged an error. That's the problem.

RAG doesn't fail loudly. It fails silently, with authority, while every dashboard stays green.

This module covers the full pipeline — ingest → chunk → embed → index → retrieve → assemble → generate — with evaluation wired in from the start, not added as an afterthought. By the end you'll have a minimal working system and, more importantly, the instrumentation to know when it's not working.

What RAG actually is

The core idea is simpler than the hype suggests. You have a large collection of documents your model doesn't know about — internal support docs, policy PDFs, a product knowledge base. Instead of retraining the model on that data (expensive and goes stale instantly), you retrieve relevant passages at query time and stuff them into the prompt.

flowchart LR
    D[Documents] --> P[Parse]
    P --> C[Chunk]
    C --> E[Embed]
    E --> I[(Vector Index)]
    Q[User query] --> EQ[Embed query]
    EQ --> R[ANN search]
    I --> R
    R --> A[Assemble prompt]
    A --> G[Generate answer]
    style D fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f
    style G fill:#a855f7,stroke:#a855f7,color:#fff

Two separate pipelines share one index. The ingestion pipeline runs offline (or on a schedule): parse documents, cut them into chunks, embed each chunk, store in a vector database. The retrieval pipeline runs online, per query: embed the query, find the nearest chunks, build a prompt, generate the answer.

This separation is also where the first production gotcha hides: the index must be kept in sync with the source documents. If your ingestion pipeline runs once at launch and never again, your index is already stale.

{"type": "rag-pipeline", "query": "policy", "title": "The full RAG pipeline, step by step"}

The pipeline in code

Before talking about what breaks, build the thing. Here is a minimal but real pipeline using the OpenAI SDK (as of mid-2026) and Qdrant as the vector store. This runs.

# pip install openai qdrant-client tiktoken

import tiktoken
from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
import uuid

client = OpenAI()
qdrant = QdrantClient(":memory:")  # swap for host= in production

EMBED_MODEL = "text-embedding-3-small"
EMBED_DIM = 1536
CHAT_MODEL = "gpt-4o-mini"
COLLECTION = "docs"

# ── Ingestion ──────────────────────────────────────────────────────────────────

def chunk_text(text: str, max_tokens: int = 400, overlap: int = 50) -> list[str]:
    enc = tiktoken.encoding_for_model("gpt-4o")
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunks.append(enc.decode(tokens[start:end]))
        start += max_tokens - overlap
    return chunks

def ingest(documents: list[str]) -> None:
    if qdrant.collection_exists(COLLECTION):
        qdrant.delete_collection(COLLECTION)
    qdrant.create_collection(
        COLLECTION,
        vectors_config=VectorParams(size=EMBED_DIM, distance=Distance.COSINE),
    )
    chunks = [chunk for doc in documents for chunk in chunk_text(doc)]
    # One batched call, not one per chunk (batch in groups of ~2,000 for real corpora)
    resp = client.embeddings.create(model=EMBED_MODEL, input=chunks)
    points = [
        PointStruct(id=str(uuid.uuid4()), vector=d.embedding, payload={"text": chunk})
        for chunk, d in zip(chunks, resp.data)
    ]
    qdrant.upsert(COLLECTION, points=points)

# ── Retrieval + Generation ─────────────────────────────────────────────────────

def retrieve(query: str, top_k: int = 5) -> list[str]:
    resp = client.embeddings.create(model=EMBED_MODEL, input=query)
    qvec = resp.data[0].embedding
    hits = qdrant.query_points(COLLECTION, query=qvec, limit=top_k).points
    return [h.payload["text"] for h in hits]

def answer(query: str) -> str:
    chunks = retrieve(query)
    context = "\n\n---\n\n".join(chunks)
    system = (
        "You are a helpful assistant. Answer using ONLY the provided context. "
        "If the context does not contain the answer, say so explicitly."
    )
    user = f"Context:\n{context}\n\nQuestion: {query}"
    resp = client.chat.completions.create(
        model=CHAT_MODEL,
        messages=[{"role": "system", "content": system}, {"role": "user", "content": user}],
        temperature=0,
    )
    return resp.choices[0].message.content

This is the baseline. It works in a demo. Now let's understand why it fails in production, and what to do about each failure.

Chunking: the highest-variance decision you'll make

Everything downstream — embedding quality, retrieval precision, generation faithfulness — depends on how you cut the documents. Choosing chunk size and strategy is not a one-time config decision; it is an experiment you run and measure.

Fixed-size chunking (what the code above does) is fast and predictable, but cuts across sentence boundaries. A chunk ending mid-sentence, mid-table, or mid-list item produces a fragment that embeds poorly and confuses the reader who sees it in the generated answer.

256–512 tokens is the empirical sweet spot for most prose corpora. Below 100 tokens, the chunk carries too little reasoning signal — it might be one sentence that doesn't stand alone. Above 1,000 tokens, the embedding has to average over multiple topics, so the cosine similarity search becomes imprecise.

For better boundary detection, use a recursive splitter that tries paragraph breaks, then sentence breaks, then word breaks before falling back to character splits. LangChain's RecursiveCharacterTextSplitter implements this. It is a practical improvement with near-zero additional cost.

For higher retrieval quality, Contextual Retrieval (Anthropic, September 2024) prepends a 50–100 token LLM-generated summary to every chunk before embedding. "This chunk describes the refund policy for digital goods purchased between January 2023 and March 2024." Retrieval failure rates drop ~49% standalone, ~67% combined with a reranker. The cost is roughly $1.02 per million document tokens — only viable if you use prompt caching, which drops it from an order of magnitude higher. See the full treatment in Chunking Strategies That Actually Matter.

Parent-child chunking stores small chunks in the index (for precise retrieval) but returns the larger parent block to the LLM (for better coherence). Index 150-token chunks; when one matches, return its 600-token parent. This is what LlamaIndex calls SentenceWindowNodeParser.

Retrieval: why vector search alone isn't enough

Dense vector search finds semantically similar chunks. It's good at paraphrase, synonyms, and conceptual matching. It is bad at exact matches: product SKU B07XZ-QK9L, error code ERR_SSL_PROTOCOL_ERROR, a person's full name. The model that generated your embeddings has never seen your internal error codes. They land in a random part of embedding space.

BM25 — the classic term-frequency inverse-document-frequency inverted index search — handles exact terms natively. It's what Elasticsearch and OpenSearch use under the hood. It has no concept of "similar meaning," but for product codes and proper nouns it is unbeatable.

Hybrid search combines both: run BM25 and dense retrieval independently, get two ranked lists, merge them with Reciprocal Rank Fusion (RRF). RRF scores each document as 1 / (rank + 60) and sums across both lists. No tuning required. A 60/40 dense/sparse split is a solid default; Qdrant, Weaviate, and OpenSearch all support this natively as of 2025.

{"type": "hybrid-search", "alpha": 0.6, "title": "BM25 + dense retrieval fused with RRF"}

Add a cross-encoder reranker for the final pass. A bi-encoder (used for vector search) encodes query and document independently, which is fast but loses interaction signals. A cross-encoder scores the query and document jointly — much more accurate, but ~50–200ms slower per call. The practical pattern: retrieve 20 candidates with hybrid search, rerank to top 5 with a cross-encoder. Cohere Rerank, BGE-reranker, and FlashRank are the production options as of mid-2026.

The multi-stage pipeline, end to end:

flowchart LR
    Q[Query] --> BM25[BM25 search<br/>top 20]
    Q --> DENSE[Dense ANN<br/>top 20]
    BM25 --> RRF[RRF fusion<br/>top 20 merged]
    DENSE --> RRF
    RRF --> RERANK[Cross-encoder<br/>rerank → top 5]
    RERANK --> ASSEMBLE[Prompt assembly]
    style Q fill:#00e5ff,color:#0a0a0f
    style RERANK fill:#a855f7,color:#fff

This stack — BM25 + dense + RRF + cross-encoder rerank — is the 2025–2026 production consensus. Starting with vector-only retrieval isn't wrong for a prototype, but you should expect to upgrade.

The top-k lie

You set top_k=5 and assume the model will use all five chunks. That's not what happens.

Lost in the Middle is a documented phenomenon: LLMs pay strong attention to content at the very beginning and end of their context window, and much less to what falls in the middle. When you assemble a prompt with five retrieved chunks stacked sequentially, the chunk that lands in position 3 receives far less attention than chunks 1 and 5 — even if chunk 3 is the most relevant. Measured accuracy drops 30%+ for relevant content in the center position.

The fix is to reorder: sort retrieved chunks by relevance score descending, then alternate between placing the most relevant at the start and end of the context block, with less relevant chunks in the middle. LangChain ships this as LongContextReorder; most teams just call it lost-in-the-middle reordering.

There's a subtler gap between demo and production hiding in the queries themselves. Your demo queries were full sentences written by someone who knew what the corpus contained. Real users type "sso not working" — or, three turns into a conversation, "what about the enterprise tier?", a query that embeds to nothing useful without the preceding turns. The standard fix is query rewriting: before retrieval, have a cheap model condense the conversation into one standalone, fully specified query ("How do I configure SAML SSO on the enterprise tier?"). It costs one extra small-model call per turn and is the cheapest fix with the biggest payoff for conversational RAG.

Also: not every question can be answered from a single retrieval pass. Questions like "How did our refund policy change between Q1 2023 and Q1 2024?" require combining information from multiple documents. Single-shot RAG fails these. For multi-hop questions, look at Agentic RAG: When Retrieval Becomes a Reasoning Loop, which covers structured retrieval loops that fetch additional context when the first pass falls short.

Evaluation from day one

The single biggest mistake engineers make with RAG is shipping without instrumentation. You discover the problems from user complaints, not from your own dashboards.

Wire two metrics from the start:

Retrieval hit-rate: for a set of test questions where you know the answer lives in document D, what fraction of the time does the retriever actually return a chunk from D in the top-k results? This is your leading indicator. If hit-rate is 70%, answer correctness is capped at 70% before the generator even starts — no prompt engineering recovers a chunk that was never retrieved. Measure this on a golden set of 50–100 question/source-document pairs.

Faithfulness: of the claims in the generated answer, what fraction are actually supported by the retrieved context? RAGAS computes this without ground-truth labels by using an LLM to check each claim against the retrieved chunks. Low faithfulness is almost always a retrieval problem: the model is filling gaps because the right chunks weren't there.

# pip install ragas

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
from datasets import Dataset

samples = {
    "user_input": ["What is our return policy for electronics?"],
    "response": ["Electronics can be returned within 30 days..."],
    "retrieved_contexts": [["Our electronics return policy allows 30-day returns..."]],
    "reference": ["Electronics may be returned within 30 days of purchase."],
}
result = evaluate(Dataset.from_dict(samples), metrics=[faithfulness, answer_relevancy])
print(result)

RAGAS scores can be gamed — a retriever that always returns very long chunks scores high on Context Recall but isn't actually a good retriever. So run automated RAGAS scoring on your golden set to catch regressions in CI, then validate monthly with 50 hard adversarial cases reviewed by a human.

The full evaluation stack is covered in RAG Evaluation with Ragas and Eval First: Building the Safety Net Before You Need It.

What breaks in practice

Most production RAG failures fall into four categories, and they're not the ones people expect.

Stale index. Your ingestion pipeline ran once at launch. A policy document changed last Tuesday. The index doesn't know. The LLM confidently cites the old version because that's what's in the index. Fix: treat index freshness as a first-class metric. Log document_last_updated and chunk_indexed_at per chunk. Alert when the gap exceeds your staleness budget.

Distractor chunks. At demo time you had 20 documents. Production has 8,000. Now a query about "refund policy for subscription products" retrieves three chunks about physical product returns, one about international shipping, and one actually relevant chunk. The LLM hedges, blends, and occasionally hallucinates a synthesis of the four wrong chunks. This is why retrieval precision matters more than raw hit-rate at scale. Hybrid search + reranking helps. Metadata filtering — restricting retrieval to documents matching the query's category, date range, or product line — helps more.

Questions that need joins. "What changed in our pricing between last year and this year?" requires two chunks from two documents that may have no embedding similarity to each other. Flat vector search cannot solve this. Options: query expansion (run two separate queries and merge the results), GraphRAG for corpora that have explicit entity relationships, or agentic retrieval that issues follow-up queries when the first pass is insufficient.

Hallucination on top of correct retrieval. Even when the right chunk is present, the model can confabulate. A Stanford study found legal AI tools hallucinate 17–33% of the time even with RAG enabled. The instruction "answer using ONLY the provided context, say explicitly if you cannot answer" helps. Structured outputs that require citing a source_chunk_id per claim help more — you can validate citations programmatically. The comprehensive treatment is in Hallucination Mitigation and Output Validation.

flowchart TD
    FAIL[RAG failure mode] --> S[Stale index]
    FAIL --> D[Distractor chunks]
    FAIL --> J[Questions needing joins]
    FAIL --> H[Hallucination on correct context]
    S --> FS[Track document freshness, alert on lag]
    D --> FD[Hybrid search plus metadata filtering]
    J --> FJ[Query expansion or agentic retrieval]
    H --> FH[Cite sources, validate citations programmatically]
    style FAIL fill:#ff2e88,color:#111
    style FS fill:#0e7490,color:#fff
    style FD fill:#0e7490,color:#fff
    style FJ fill:#0e7490,color:#fff
    style FH fill:#0e7490,color:#fff

RAG vs. long context vs. fine-tuning

The question comes up in every team that gets past the demo stage. Here is the decision in plain terms.

Long-context inference costs roughly 20–24x more than RAG at production volume (illustrative as of mid-2026, based on per-token pricing differentials). For a one-time analysis task during development — "summarize this 200-page contract" — just drop it in context. For a production system handling 10,000 queries per day over a 50,000-document knowledge base, the math breaks fast. Accuracy also degrades: multiple 2025 evaluations showed 20%+ accuracy drops on knowledge-retrieval tasks as context length grew past the point where the relevant fact was buried.

Fine-tuning changes how the model behaves, not what it knows. If you need the model to always respond in a specific JSON format, to follow a tool-calling protocol exactly, or to maintain a consistent tone — fine-tune. If you need it to know that your refund window is 45 days — use RAG. Fine-tuning on facts leads to hallucinated facts on out-of-distribution questions, because the model is pattern-matching on training distribution rather than looking up stored text. See Fine-Tuning Decision Framework for when fine-tuning is actually justified.

The correct default for any knowledge-grounding task: RAG first, evaluate, then add fine-tuning on top if behavioral issues persist.

{"type": "context-window", "window": 128000, "title": "How retrieved chunks consume your context budget"}

A real cost envelope

Before you reach for the expensive stack, sketch the numbers. A production RAG system at modest scale (1,000 queries/day):

Embedding (text-embedding-3-small, ~$0.02/million tokens):
  Query embeddings: 1,000 queries × 100 tokens avg = 100K tokens/day ≈ $0.002/day
  Ingestion (one-time): 10,000 chunks × 400 tokens = 4M tokens ≈ $0.08 total

Reranking (Cohere Rerank API, ~$1/thousand calls as of mid-2026):
  1,000 queries × rerank 20 to 5 = 1,000 calls/day ≈ $1/day

Generation (gpt-4o-mini, ~$0.15/million input tokens):
  1,000 queries × (5 chunks × 400 tokens + 200 token question) = ~2.2M input tokens/day
  ≈ $0.33/day in generation input costs

Total: roughly $1.40/day at 1,000 queries, dominated by reranking.
Scale to 10,000 queries: ~$14/day, ~$420/month.

This is why RAG is the economically correct default over long-context. Start with the strongest argument: the corpus above — 10,000 chunks × 400 tokens = 4M tokens — does not fit in any context window, so for a knowledge base at that scale there is no long-context option to price. Now shrink the corpus until it does fit: ~200K tokens (a few hundred documents), inside a modern 200K–1M window. Stuffing all of it into every prompt at 1,000 queries/day is 1,000 × 200K = 200M input tokens/day ≈ $30/day at $0.15/million — versus ~$1.40/day for the entire RAG stack, a bit over 20x. Prompt caching narrows the gap (at a 50% cached-input discount, illustrative, that's ~$15/day, still ~10x) but doesn't close it, and the cache invalidates every time a document changes. The reranker is the swing cost at moderate scale; it's often worth it for precision, but if latency is tight you can skip it for a first release and add it once you've measured the precision gap.

Where to go next

The crash course has covered the end-to-end shape. Each component has its own depth.

• Embeddings and Vector Databases That Scale — how ANN indexes actually work, HNSW vs. IVF, and when pgvector is enough
• Chunking Strategies That Actually Matter — late chunking, contextual retrieval, parent-child indexing, and the benchmarks behind them
• Hybrid Search: BM25 Plus Dense Retrieval Plus Reranking — full treatment of SPLADE, ColBERT, RRF tuning, and cross-encoder selection
• RAG Evaluation with Ragas — wiring RAGAS into CI, building adversarial golden sets, and avoiding metric gaming
• RAG Evaluation, Failure Modes, and the Decision Between RAG, Long Context, and Fine-Tuning — the full decision framework with numbers
• Agentic RAG: When Retrieval Becomes a Reasoning Loop — when single-shot retrieval isn't enough, and how to prevent retrieval thrash
• Context Engineering: Filling the Window Right — how the retrieval output fits into a larger context budget that includes system prompts, tools, and conversation history