RAG evaluation, failure modes, and when to use long context or fine-tuning instead

The support chatbot passed every test before launch. On day one it handled five hundred queries. On day six, the first complaint landed: a user had been confidently quoted a refund window that expired eight months ago. The retriever found a chunk mentioning 'refund,' the LLM wrote a fluent answer, and the automated eval suite gave it a green checkmark. The pipeline worked exactly as designed — it retrieved the wrong thing and said it with authority.

This is the evaluation problem in RAG. Fluency is not correctness, and "the model answered" is not the same as "the model answered faithfully from current knowledge." Without measurement at each layer of the pipeline, you have no idea which part broke, and no principled basis for choosing which fix to try next.

The four metrics that actually matter

RAGAS (Retrieval Augmented Generation Assessment, docs) provides four metrics that decompose RAG quality into diagnosable signals. Three of them are reference-free — no human-labeled ground-truth answers; an LLM judge scores them, making them practical to run in CI without a labeled dataset. The fourth, Context Recall, needs at least a partial reference answer.

Faithfulness is the most operationally important metric. It measures what fraction of claims in the generated answer are supported by the retrieved context. A faithfulness score of 0.6 means 40% of the answer's claims could not be traced to any retrieved chunk — those are potential hallucinations or knowledge injected from the model's pretraining weights. When faithfulness is low, check Context Precision and Recall before blaming the model: if retrieval starved it, the LLM filled the gaps and the fix is in retrieval; if the context was good and the model ignored it, the fix is a tighter generation prompt. Watch the inverse trap too — a retriever that returns the wrong chunk, faithfully quoted, scores high on faithfulness. The stale refund answer in the opening anecdote was perfectly faithful to its retrieved chunk. Faithfulness tells you about grounding, not about whether the grounding was right.

Context Precision measures whether the retrieved chunks are actually relevant to the query. A retriever that pulls 10 chunks and only 3 are on-topic scores around 0.3. Low precision wastes context window budget and dilutes the signal that matters. The classic cause is an embedding model that conflates semantic similarity with topical relevance — it retrieves everything about 'bank account' when you ask about 'river bank erosion.'

Context Recall measures how much of the information needed to answer the question actually appeared in the retrieved chunks. This one requires at least a partial reference answer to compute properly. Low recall means the retriever missed documents that contained the right answer — a chunking problem (the answer was split across a chunk boundary), an embedding model mismatch, or a vocabulary gap that hybrid search with BM25 would have caught.

Answer Relevancy measures how directly and concisely the answer addresses the question. It penalizes verbosity and topic drift. A model that always hedges with paragraphs of "It depends" and "There are many factors to consider" before reaching the point will score low here even if the underlying information is correct.

flowchart LR
    CONTEXT["Retrieved chunks"] --> FAITH["Faithfulness\nAre answer claims\nsupported by context?"]
    QUERY["User query"] --> CREC["Context Recall\nIs needed info\nin the context?"]
    QUERY --> CPREC["Context Precision\nAre retrieved chunks\nactually relevant?"]
    CONTEXT --> CPREC
    QUERY --> AREL["Answer Relevancy\nDoes the answer\naddress the query?"]
    CONTEXT --> CREC
    REF["Reference answer\n(partial is enough)"] --> CREC
    ANSWER["Generated answer"] --> FAITH
    ANSWER --> AREL
    style FAITH fill:#00e5ff,color:#0a0a0f
    style CREC fill:#0e7490,color:#fff
    style CPREC fill:#0e7490,color:#fff
    style AREL fill:#15803d,color:#fff
    style REF fill:#ffaa00,color:#111

The practical evaluation loop is: generate 50–100 test questions by prompting an LLM over your actual documents, manually verify 10–15 of them look realistic and hard, then run RAGAS against a snapshot of your pipeline. Score all four metrics. When you see faithfulness below 0.7, look at the retriever before touching the generation prompt.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Build this from your actual pipeline runs
eval_data = {
    "question": ["What is our return window for electronics?", ...],
    "answer": ["Our return window for electronics is 30 days...", ...],
    "contexts": [["chunk 1 text", "chunk 2 text"], ...],
    "ground_truth": ["Electronics returns are accepted within 30 days...", ...],
}

dataset = Dataset.from_dict(eval_data)
result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# {'faithfulness': 0.82, 'answer_relevancy': 0.91,
#  'context_precision': 0.74, 'context_recall': 0.68}

DeepEval is a heavier alternative with a more granular metric suite and native support for custom LLM judges. It is better for multi-turn evaluation and when you need assertion-level detail on which specific claims failed. RAGAS is simpler to get running in a CI pipeline; DeepEval gives more diagnostic signal for a dedicated evaluation harness.

The RAGAS gaming problem

Here is the trap: automated metrics can be gamed without intending to. A retriever configured to return very long, verbose chunks scores high on Context Recall — the needed information is almost always in there somewhere if you return 5,000 tokens per query. But that retriever is not actually good; it is burying the relevant signal in noise and blowing up your context budget. Context Precision will reveal this, but only if you check both metrics together.

The defense is a small but genuine hard-case evaluation set: 50–100 adversarial questions where naive retrieval reliably fails — questions that use different vocabulary than the documents, questions where the answer spans multiple chunks, questions where a plausible but wrong chunk exists. Score these by hand after every major pipeline change. Your automated metrics tell you the average-case trend; your hard-case set tells you whether you broke the edge cases.

{ "type": "eval-pipeline", "title": "RAG evaluation safety net: where each metric fires" }

What actually breaks in production

RAG has a specific failure-mode profile. Understanding which failure mode you have determines which fix to try.

Retrieval failure: wrong chunks

The most common failure is retrieval returning chunks that are topically adjacent but not actually relevant. A question about software deployment returns chunks about software architecture because they share vocabulary. Hybrid search — BM25 for exact term matching plus dense retrieval for semantics — catches most of this. If your pipeline does pure dense retrieval only, adding BM25 is the highest-ROI fix available, discussed in depth in Hybrid Search: BM25 Plus Dense Retrieval Plus Reranking.

Reranking catches what hybrid search misses: the recall stage returns 20 candidates, and a cross-encoder reranker (Cohere Rerank, BGE-reranker, FlashRank) scores query-passage pairs jointly to promote the actual best matches to the top 5. This adds 50–200ms of latency. HyperRAG (2025) introduced KV-cache reuse across reranker calls, cutting that tax significantly in high-throughput settings.

Lost in the middle

This one is counterintuitive and underappreciated. Liu et al. documented it in "Lost in the Middle": LLMs pay disproportionate attention to content at the very start and very end of their context window. Content sitting in the center gets significantly less attention. When your retriever returns 20 chunks and the most relevant one lands in position 10, accuracy can drop 30% or more compared to when that same chunk is first or last.

The fix is straightforward: after reranking, reorder chunks so the highest-scored ones appear at positions 1 and N (the last), and lower-scored chunks fill the middle. Most retrieval pipelines return chunks in score order and never think about this. One sort step cuts the lost-in-the-middle penalty substantially.

{ "type": "lost-middle", "title": "How chunk position degrades LLM accuracy" }

Hallucination on top of correct retrieval

A Stanford RegLab/HAI study of legal AI tools (Magesh et al.) found hallucination rates of 17–33% even with RAG enabled. Retrieval reduces the surface area for hallucination but does not eliminate it. The LLM still generates from pretraining weights when the retrieved context is ambiguous or incomplete. When faithfulness is low despite correct chunks being retrieved, the fix is on the generation side: tighter system prompt instructions ("answer only from the provided context, say 'I don't know' if the context is insufficient"), smaller context windows, or a self-consistency check where the answer is verified against the retrieved context in a second pass.

Constraining generation too aggressively creates a different problem: the model hedges everything and refuses to synthesize across chunks, producing unhelpful "I cannot determine this from the provided context" responses for questions the context clearly does answer. The practical balance is a system prompt that explicitly permits synthesis and inference while requiring that every factual claim be traceable to a specific retrieved chunk.

Agentic-specific failures

Agentic RAG introduces three failure modes that flat RAG does not have:

Retrieval thrash: the agent loops, retrieving the same documents repeatedly across iterations because it has no memory of what it has already retrieved. Fix: a deduplication cache keyed on chunk ID at the agent level.

Tool storms: the agent calls the retrieval tool far more often than necessary per reasoning step, burning latency and tokens without accumulating new signal. Fix: max-retrieval-calls-per-step limits.

Context bloat: retrieved chunks accumulate across a long agent run and eventually fill the context window, degrading reasoning quality. Fix: periodic context compression — summarize earlier retrieved chunks and replace them with the summary before continuing.

All three require you to add budgets and guardrails at the orchestration layer, not the retrieval layer.

The numbers that drive the RAG vs. long context decision

The "long-context models make RAG obsolete" claim circulates at every conference. The cost math does not support it for most production use cases.

Cost comparison (illustrative, mid-2026):

RAG pipeline:
  Input tokens per query: ~2,000 (system prompt + question + 3–5 retrieved chunks)
  Output tokens per query: ~300
  At frontier model pricing: ~$0.001–$0.003 per query
  10M queries/month: $10,000–$30,000

Full-context approach (50,000-token knowledge base in every prompt):
  Input tokens per query: ~50,300 (full KB + system + question)
  At 25x the tokens: ~$0.025–$0.075 per query
  10M queries/month: $250,000–$750,000

Same approach with prompt caching (cached input at ~10% of list price):
  ~$0.004–$0.010 per query → 10M queries/month: $40,000–$100,000
  Gap vs. RAG shrinks from ~25x to roughly 3–4x on input cost

Break-even, uncached: full-context wins only when your knowledge base fits in
< ~4,000 tokens — roughly three pages of text. With a reliably warm cache the
break-even stretches to tens of thousands of tokens, and the deciding factor
shifts from token price to accuracy degradation and cache churn.

Prompt caching is the strongest counterargument to this math, so run it honestly. A static knowledge base re-sent in every prompt is the ideal cache workload — the same mechanism that makes Contextual Retrieval's indexing cost $1.02 per million tokens in the section below. But caches have write costs (~25% over list price on the first send), TTLs measured in minutes unless you pay to extend them, and any edit to the corpus invalidates everything after the changed prefix. A knowledge base that updates daily busts the cache daily; one that grows per-query never caches at all. RAG still wins for large or changing corpora, but when the cache holds, the margin is 3–4x rather than 25x — and the accuracy argument below carries more of the weight.

The accuracy story is also not what the long-context marketing suggests. Long-context benchmarks consistently show double-digit accuracy degradation as context grows from a few thousand to 100,000+ tokens on multi-document QA tasks. The lost-in-the-middle effect discussed above applies at scale too: a 128k-token context window is not uniformly attended to. You can tune retrieval to return exactly the right 2,000 tokens; you cannot tune attention to focus on the right 2,000 tokens out of 128,000.

Long context has real advantages in specific scenarios. For single-document analysis — asking questions about one long PDF — you want the full document in context rather than fragmenting it with chunking. For prototyping and debugging, loading your small test corpus directly eliminates the retrieval failure mode as a variable. For tasks that require cross-document synthesis over a small, stable set of documents, the retrieval overhead may not be worth it.

See Long Context vs RAG: The Million-Token Question for the full treatment of this tradeoff; the focus here is on the cost signal and its implication for when to default to RAG.

The RAG vs. fine-tuning decision

This is the one teams get wrong most consistently. Fine-tuning and RAG are not competing solutions to the same problem.

RAG changes what the model knows. Private documents, dynamic facts, data that changes daily, customer records — none of this can go into a fine-tuning dataset without going stale. The right answer to "how do I get the model to know our Q3 pricing?" is RAG, always.

Fine-tuning changes how the model behaves. Output format, tone, domain-specific reasoning conventions, structured classification schemas, tool-calling policies — these are behavioral patterns that are stable over time and do not depend on current facts. A fine-tuned model for medical coding learns to apply ICD-10 coding conventions consistently; that behavior is what changed, not the facts.

flowchart TD
    PROBLEM["Production problem"] --> Q1{"Is it about\nwhat the model knows\nor how it behaves?"}
    Q1 -->|"Wrong or stale facts,\nknowledge not in pretraining"| RAG["Use RAG\n(or update retrieval pipeline)"]
    Q1 -->|"Wrong format, tone,\nreasoning pattern,\nclassification"| FINEQ{"Is the failure\nconsistent and\ndataset-feasible?"}
    FINEQ -->|"Yes — stable pattern\nwith examples"| FT["Consider fine-tuning"]
    FINEQ -->|"No — variable,\nfew examples"| PROMPT["Fix the prompt\nor system instructions"]
    RAG --> BOTH["RAG + fine-tuning:\nuse when behavior\nAND knowledge are wrong"]
    FT --> BOTH
    style RAG fill:#0e7490,color:#fff
    style FT fill:#00e5ff,color:#0a0a0f
    style PROMPT fill:#15803d,color:#fff
    style BOTH fill:#a855f7,color:#fff

The failure mode of fine-tuning on facts: you bake in today's pricing, tomorrow it changes, and the model confidently quotes the wrong number. The knowledge is now in weights, not in a retrievable document you can update. A detailed treatment of the fine-tuning decision framework is in the Fine-Tuning section of the course.

Fine-tuning and RAG are frequently the right combination: fine-tune for domain reasoning style, use RAG to supply current knowledge. A legal AI assistant might be fine-tuned to reason like a lawyer (structured issue spotting, citation conventions, hedged conclusions) while relying on RAG for current case law and specific contract clauses.

What Contextual Retrieval adds and what it costs

Anthropic's Contextual Retrieval (September 2024) is worth covering here because it directly improves Context Recall — the metric most teams see degrade when chunks lose surrounding context.

The mechanic: before embedding each chunk and before BM25 indexing, call an LLM to generate a 50–100 token summary of how the chunk relates to its parent document, then prepend that summary to the chunk. This means a chunk about "Section 7 cancellation clauses" also carries the context that it appears in a service agreement template for enterprise customers. Retrieval queries about "how can I cancel my enterprise subscription" now match on meaning rather than just surface keywords.

Retrieval failures dropped 49% in Anthropic's own evaluations, and 67% when combined with reranking. The cost with prompt caching enabled runs approximately $1.02 per million document tokens — a one-time indexing cost that amortizes across all future queries against that corpus. Without prompt caching, the cost is an order of magnitude higher, making it impractical for large corpora that change frequently.

The practical guidance: apply Contextual Retrieval on static or slowly-changing corpora (legal documents, product manuals, policy archives). Skip it for corpora that update daily unless you can limit context generation to changed documents only.

What breaks (the failure modes in one place)

RAG fails in a predictable taxonomy. Here is the full list with the diagnosis and fix for each:

Failure	Signal	Root cause	Fix
Wrong chunks retrieved	Context Precision low	Vocabulary mismatch, no BM25, poor chunk size	Add hybrid search; tune chunk size to 256–512 tokens
Needed info missing	Context Recall low	Chunk boundary split the answer; embedding model gap	Contextual Retrieval or smaller chunks with overlap
Answer contradicts context	Faithfulness low	LLM ignoring context, filling from pretraining	Tighter system prompt; verify context contains the answer
Off-topic or verbose answer	Answer Relevancy low	Prompt too loose; context noise distracting generation	Constrain output format; reduce top-k chunks
Relevant chunk ignored	Accuracy drops despite correct retrieval	Lost in the middle	Reorder: highest-relevance chunks at start and end
Stale answer	Factual errors on dynamic data	Knowledge base not refreshed	Ingestion pipeline freshness; TTL on embeddings
Retrieval loop in agent	Latency spikes; same docs retrieved N times	No deduplication state	Chunk ID dedup cache per agent run
Fine-tuning didn't fix stale facts	New facts still wrong post-training	Wrong tool for the problem	Switch to RAG for the dynamic facts

Building the evaluation pipeline

A production-grade RAG evaluation pipeline has three layers:

Offline (pre-deploy): Run RAGAS on your golden question set in CI. A RAGAS faithfulness drop of more than 0.05 from the baseline blocks the deploy. This catches retrieval regressions from chunking changes, embedding model updates, or prompt drift.

Pre-release adversarial sweep: A set of 50–100 hard-case questions evaluated manually before major releases. These target known failure modes: cross-document synthesis questions, questions using synonyms not in the indexed text, questions where plausible-wrong chunks exist.

Online (production sampling): Sample 1–3% of live queries, run RAGAS asynchronously, and track the metrics trend. Alert on sudden drops in faithfulness or context precision — these signal upstream data quality problems (newly ingested malformed documents, schema changes breaking chunkers) before users notice.

Two costs people wave away. First, LLM-judged metrics are noisy: judge model choice, sampling, and question-set composition can each move faithfulness scores by several points run to run, so a 0.05 CI gate flakes unless you pin the judge model and the question set, measure run-to-run variance on an unchanged pipeline, and set the gate at 2–3x that noise floor. Second, the online layer has a bill: 1–3% of 10M queries is 100,000–300,000 judged responses a month, at several LLM calls each — budget it like any other inference workload. Put deterministic retrieval metrics (recall@k, MRR, nDCG against a small labeled retrieval set) underneath RAGAS as the cheap, noise-free first CI layer; they catch most retrieval regressions for the cost of a unit test.

The eval pipeline visualizer in the RAG Evaluation with Ragas deep-dive shows how to wire this up with Langfuse or Braintrust for the online monitoring layer.

from anthropic import Anthropic
from ragas.metrics import faithfulness

def evaluate_rag_response(question: str, answer: str, contexts: list[str]) -> dict:
    """Minimal RAGAS eval for inline checking during development."""
    from datasets import Dataset
    from ragas import evaluate

    dataset = Dataset.from_dict({
        "question": [question],
        "answer": [answer],
        "contexts": [contexts],
    })

    # faithfulness is reference-free; RAGAS's default context_precision needs
    # a reference answer, so it stays in the offline CI run with the golden set
    result = evaluate(dataset, metrics=[faithfulness])
    return {"faithfulness": result["faithfulness"]}

# In your RAG loop:
client = Anthropic()
question = "What is the cancellation policy for enterprise plans?"

# ... retrieval happens here ...
retrieved_chunks = ["Enterprise plans may be cancelled with 30 days notice...", "..."]
context_block = "\n\n".join(retrieved_chunks)

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    system="Answer based only on the provided context. If the context is insufficient, say so.",
    messages=[{
        "role": "user",
        "content": f"Context:\n{context_block}\n\nQuestion: {question}"
    }]
)
answer = message.content[0].text

scores = evaluate_rag_response(question, answer, retrieved_chunks)
if scores["faithfulness"] < 0.7:
    # Flag for review, log for the hard-case dataset
    print(f"Low faithfulness: {scores['faithfulness']:.2f} — review this response")

The decision in practice

The framework is simple once you have the metrics:

Faithfulness below 0.7: Retrieval is returning wrong chunks or the generation is ignoring the context. Check Context Precision first. If precision is fine but faithfulness is still low, tighten the generation prompt to require context citation. If precision is low, add reranking or hybrid search.

Context Recall below 0.6: The right information is not making it into the context. Chunking is usually the culprit — the answer spans a chunk boundary. Try smaller chunks with overlap, or Contextual Retrieval if your corpus is stable enough to justify the indexing cost. See Chunking Strategies That Actually Matter for the full decision tree.

Good metrics but wrong behavior: The model knows the facts but uses them wrong — formats output badly, misses domain-specific reasoning conventions, fails to apply policy rules correctly. This is the fine-tuning signal. Collect 500–2,000 examples of correct behavior, fine-tune on them, re-evaluate. Do not touch the retrieval pipeline.

Cost pressure at scale: If your knowledge base is large and dynamic, RAG is not negotiable on cost grounds. If it is small and static (a few hundred documents), long context simplifies the pipeline and may be cheaper at low query volume — specifically below ~50,000 queries per month, where the savings from retrieval infrastructure do not yet exceed the cost difference.

Multi-hop questions that flat RAG fails: The answer requires combining information across multiple documents in ways a single retrieval step cannot capture. This is the GraphRAG and structured retrieval territory — a substantially higher-complexity and higher-cost path that is only justified when multi-hop reasoning is the core requirement, not a corner case.

The default stack in 2025–2026 for a production RAG system with measurable quality requirements: hybrid search with reranking, Contextual Retrieval for static knowledge bases, RAGAS in CI, a small adversarial golden set reviewed manually before every major release, and an online monitoring sample. Fine-tuning when behavior is wrong. Long context never as the default, always as a deliberate cost-conscious choice for a specific scenario.

RAG is not a magic bullet. A Stanford study found legal AI tools hallucinate 17–33% of the time even with RAG. But unmeasured RAG — deployed without faithfulness monitoring, without a golden test set, without retrieval metrics — is worse still. You get the failures without any signal for where to fix them.

Measure it. Decompose it. Fix the layer that's actually broken.