RAG Evaluation with Ragas: Measuring Retrieval and Generation Separately

You shipped a RAG system. The product demo looked clean. Three weeks later a user pastes in a screenshot: the chatbot confidently cited a return policy that was retired eight months ago and is nowhere in your document store. You check the retrieval logs — the right chunks were retrieved. The model just ignored them and free-associated from training data. Your end-to-end quality score, measured as "did the answer match the expected answer?", never caught this because the expected answer corpus was too sparse to cover that edge case. You had no metric that would have caught a faithfulness failure separately from a retrieval failure.

That is the practical problem Ragas solves.

Why disaggregated metrics matter

The architecture of a RAG system is a pipeline: query goes into a retriever, retrieved chunks go into a generator alongside the query, answer comes out. Failures compose differently at each stage.

If the retriever has low precision, it dumps irrelevant chunks into the context window. The generator now has to reason over noise — and even a careful model will sometimes ground its answer in an irrelevant chunk rather than acknowledging it has no useful context. If the retriever has low recall, relevant information never reaches the generator at all, and the model either says "I don't know" or, worse, fills the gap from parametric memory. If the generator has low faithfulness, it produces claims the context doesn't support regardless of how well the retriever is working.

These three failure modes can all produce bad answers. They require different fixes. End-to-end scoring conflates them — see the RAG evaluation, failure modes, and alternatives article for the full taxonomy of what goes wrong and when to prefer long-context retrieval or fine-tuning instead. Ragas exists precisely to separate the signals.

flowchart TD
    START[Low RAG quality score] --> FP{Faithfulness\nhigh?}
    FP -->|"No < 0.7"| GEN_FAIL[Generator problem:\nignoring context,\nhallucinating claims]
    FP -->|"Yes ≥ 0.7"| CP{Context\nPrecision high?}
    CP -->|"No < 0.7"| NOISE[Retriever noise:\nirrelevant chunks\npolluting context]
    CP -->|"Yes ≥ 0.7"| CR{Context\nRecall high?}
    CR -->|"No < 0.7"| MISS[Retriever miss:\nneeded chunks\nnot retrieved]
    CR -->|"Yes ≥ 0.7"| AR{Answer\nRelevance high?}
    AR -->|"No < 0.7"| DRIFT[Generator drift:\ncorrect grounding,\nwrong topic]
    AR -->|"Yes ≥ 0.7"| GOOD[System healthy\nfor this query]

    style GEN_FAIL fill:#ff2e88,color:#fff
    style NOISE fill:#00e5ff,color:#0a0a0f
    style MISS fill:#ffaa00,color:#0a0a0f
    style DRIFT fill:#a855f7,color:#fff
    style GOOD fill:#15803d,color:#fff

The four Ragas metrics

Faithfulness

Faithfulness answers: does the generated answer contradict or fabricate beyond what the retrieved context actually says?

Ragas computes this by decomposing the answer into individual atomic claims, then checking each claim against the retrieved context using an LLM judge. The score is:

faithfulness = (claims supported by context) / (total claims in answer)

An answer with five claims, four of which can be traced to specific passages in the retrieved chunks and one that is an extrapolation from training memory, scores 0.80. A perfect paraphrase of the context scores 1.0. An answer that ignores the context entirely and answers from parametric knowledge scores close to 0.

This is reference-free: you only need the answer and the retrieved context, not a ground-truth reference answer. That is what makes it practical for scoring production traffic.

The failure mode to watch: a model can achieve a high faithfulness score by being vague or saying "according to the documents..." and then hedging. Faithfulness alone doesn't tell you the answer was useful. Pair it with answer relevance.

Context precision

Context precision answers: of the chunks the retriever returned, how many were actually needed to answer the question?

The formula is a mean-average-precision variant over the ranked list, so rank matters: one relevant chunk out of five scores 1.0 if it is ranked first and 0.20 if it is ranked last. The metric rewards putting relevant chunks at the top, not merely retrieving them somewhere in the list. High noise in the context window increases the probability the generator grounds its answer in an irrelevant chunk, and it burns token budget.

Despite what the name suggests, the default context_precision metric is not reference-free: the LLM judge decides, for each chunk, whether it was useful in arriving at the reference answer — so your dataset needs a reference column. Ragas ships a reference-free variant, LLMContextPrecisionWithoutReference, which judges chunks against the generated response instead; that works for production traffic but measures something subtly different, because a hallucinated answer drags chunk verdicts with it. The cost is proportional to the number of retrieved chunks, so this metric is more expensive for systems that retrieve k=10 versus k=3.

Practical implication: if context precision is low but faithfulness is acceptable, the generator is successfully ignoring the noise. You're wasting token budget but not (yet) producing wrong answers. Fix the retriever anyway — the generator won't always be so selective.

Context recall

Context recall answers: of all the information needed to correctly answer the question, what fraction did the retriever surface?

Like default context precision, this requires reference answers — and here there is no reference-free workaround, because the whole question is whether the retrieval covered what a correct answer needs. Ragas computes it by decomposing the reference answer into atomic statements, then checking whether each statement is attributable to the retrieved context:

context recall = (reference statements found in context) / (total reference statements)

If the reference answer has four key facts and the retriever returned context containing three of them, context recall is 0.75. The fourth fact — which the generator would need to answer completely — was never retrieved.

The operational cost: you need reference answers. For most teams this means maintaining a golden dataset with human-written or carefully reviewed answers for your evaluation queries. This is the same corpus described in the golden datasets article. Context recall is where that investment pays off — it catches the retriever missing chunks entirely, a failure mode the other three metrics cannot detect.

Answer relevance

Answer relevance answers: does the response actually address the question that was asked?

Ragas computes this in an elegant reference-free way: it generates several alternative questions that the answer could be responding to, then measures cosine similarity between those generated questions and the original query. High similarity means the answer is tightly scoped to the actual question; low similarity means the answer wandered.

This catches a specific failure pattern: the generator correctly grounds its response in the retrieved context, but the retrieved context was off-topic, so the answer is faithful but useless. Without answer relevance as a separate metric, high faithfulness would mask the problem.

Setting up Ragas

Install and basic usage:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    context_precision,
    context_recall,
    answer_relevancy,
)
from datasets import Dataset

# Your RAG outputs — one row per query
data = {
    "question": [
        "What is the refund window for digital products?",
        "How do I change my billing address?",
    ],
    "answer": [
        "Digital products can be refunded within 14 days of purchase if unused.",
        "You can update your billing address in Account Settings under Payment Methods.",
    ],
    "contexts": [
        # Retrieved chunks — list of strings per query
        [
            "All digital downloads are eligible for refund within 14 days provided the item has not been accessed.",
            "Physical products have a 30-day return window.",
        ],
        [
            "To update payment information, navigate to Account Settings, click Payment Methods, then Edit.",
        ],
    ],
    # reference is required by context_precision and context_recall;
    # faithfulness and answer_relevancy ignore it
    "reference": [
        "Digital products can be refunded within 14 days if not accessed.",
        "Update billing address in Account Settings → Payment Methods → Edit.",
    ],
}

dataset = Dataset.from_dict(data)

results = evaluate(
    dataset,
    metrics=[
        faithfulness,
        context_precision,
        context_recall,
        answer_relevancy,
    ],
)

print(results)
# Output (illustrative):
# {
#   'faithfulness': 0.91,
#   'context_precision': 0.83,
#   'context_recall': 0.88,
#   'answer_relevancy': 0.94,
# }

Ragas uses OpenAI by default for its judge calls. Swap in a different model:

from ragas.llms import LangchainLLMWrapper
from langchain_anthropic import ChatAnthropic

# Use Claude as the judge — different family from your OpenAI-powered generator
# to avoid narcissistic bias inflating faithfulness scores
judge_llm = LangchainLLMWrapper(ChatAnthropic(model="claude-sonnet-4-5"))

results = evaluate(
    dataset,
    metrics=[faithfulness, context_precision, answer_relevancy],
    llm=judge_llm,
)

Choosing a judge model from a different family than your RAG generator matters. LLM-as-judge calibration research documents GPT-4 self-preference bias at ~10% and Claude narcissistic bias at ~25% when judging own-family outputs. If your RAG system uses Claude as the generator and you use Claude as the Ragas judge, faithfulness scores will be artificially inflated.

Integrating Ragas into production

CI quality gate

The minimal viable Ragas CI setup runs three metrics against a fixed golden dataset on every pull request. One rule matters more than any other: the golden file stores only questions and reference answers, and the answers and contexts are regenerated by the PR's pipeline on every run. Score stored outputs instead and the gate tests a snapshot of last month's system — every PR passes, nothing is caught, and the gate is theater.

# eval/test_rag_quality.py — runs in CI on every PR
import json
import pytest
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision, answer_relevancy
from datasets import Dataset

from app.rag import retrieve, generate  # the pipeline under test

THRESHOLDS = {
    "faithfulness": 0.75,
    "context_precision": 0.70,
    "answer_relevancy": 0.80,
}

def load_golden_questions() -> list[dict]:
    # Versioned eval set: {"question": ..., "reference": ...} per line
    # 50 examples detects major regressions; 200 gives statistical confidence
    with open("eval/golden_rag_dataset.jsonl") as f:
        return [json.loads(line) for line in f]

def test_rag_quality_thresholds():
    golden = load_golden_questions()
    # Run the PR's actual retriever and generator — never stored outputs
    rows = {"question": [], "answer": [], "contexts": [], "reference": []}
    for ex in golden:
        chunks = retrieve(ex["question"])
        rows["question"].append(ex["question"])
        rows["answer"].append(generate(ex["question"], chunks))
        rows["contexts"].append(chunks)
        rows["reference"].append(ex["reference"])

    results = evaluate(
        Dataset.from_dict(rows),
        metrics=[faithfulness, context_precision, answer_relevancy],
    )
    for metric, threshold in THRESHOLDS.items():
        score = results[metric]
        assert score >= threshold, (
            f"RAG quality regression: {metric} = {score:.3f}, "
            f"threshold = {threshold:.3f}"
        )

Keep context recall out of CI until your reference answers are complete enough to trust — it is the most sensitive metric to reference quality, and a sparse reference corpus produces recall scores that flatter the retriever. The three metrics above catch the majority of retrieval and generation regressions. See offline vs online evaluation for the full discussion of what CI catches versus what production sampling catches.

{ "type": "eval-pipeline", "title": "Where Ragas fits in the five-stage eval safety net" }

Scoring production traffic with Langfuse

Ragas v0.2+ integrates with Langfuse so scores attach to existing traces:

from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
from datasets import Dataset

langfuse = Langfuse()

@observe()
def answer_question(query: str) -> dict:
    # Your existing RAG pipeline
    chunks = retrieve(query)          # returns list[str]
    answer = generate(query, chunks)  # returns str

    # Score this individual request
    row = Dataset.from_dict({
        "question": [query],
        "answer": [answer],
        "contexts": [chunks],
    })
    scores = evaluate(row, metrics=[faithfulness, answer_relevancy])

    # Attach scores to the Langfuse trace for this request
    langfuse_context.score_current_trace(
        name="faithfulness",
        value=float(scores["faithfulness"]),
    )
    langfuse_context.score_current_trace(
        name="answer_relevancy",
        value=float(scores["answer_relevancy"]),
    )

    return {"answer": answer, "chunks": chunks}

Scoring every single production request is expensive (see the cost estimate in the summary block). Score a random sample instead — 5–10% of traffic is enough to detect meaningful drift. Set an alert when the rolling 7-day faithfulness average drops below 0.70. The tracing and observability tooling comparison covers how Langfuse, Braintrust, and Arize each handle threshold alerting.

A worked diagnostic example

Suppose your Ragas dashboard shows these numbers after a prompt change last Tuesday:

Before change:
  faithfulness:       0.88
  context_precision:  0.79
  answer_relevancy:   0.91

After change:
  faithfulness:       0.62   ← dropped 0.26
  context_precision:  0.81   ← roughly stable
  answer_relevancy:   0.89   ← roughly stable

Faithfulness dropped sharply while the retrieval metrics stayed flat. The problem is in the generator, not the retriever. The prompt change — let's say you removed an instruction to "only use the provided context" — allowed the model to draw on training data to supplement gaps in the retrieved chunks. The fix: restore the grounding constraint, or add a citation-forcing instruction.

Now a different scenario, after an update to the embedding model powering your vector search:

Before:
  faithfulness:       0.87
  context_precision:  0.76
  answer_relevancy:   0.90

After:
  faithfulness:       0.83   ← slight drop
  context_precision:  0.51   ← dropped 0.25
  answer_relevancy:   0.88   ← roughly stable

Context precision tanked. The new embedding model has lower semantic precision for your domain — it's retrieving chunks that are topically adjacent but not actually relevant. The faithfulness score held reasonably because the generator is good at staying grounded even when the context is noisy; but the token waste and latency overhead from irrelevant chunks is real, and faithfulness will eventually erode as retrieval gets noisier. Fix the retriever: either revert the embedding model, fine-tune it on domain data, or add a reranker to filter noise post-retrieval. See hybrid search: BM25 plus dense retrieval plus reranking for the reranker patterns that help most here.

sequenceDiagram
    participant CI as CI Pipeline
    participant RAGAS as Ragas Eval
    participant DB as Langfuse / Arize
    participant ENG as Engineer

    CI->>RAGAS: run 3 metrics on 200-example golden set
    RAGAS->>CI: faithfulness=0.62 (below threshold 0.75)
    CI->>CI: FAIL pull request
    CI->>ENG: PR blocked — faithfulness regression

    Note over ENG: investigate which queries failed

    ENG->>DB: query production traces from last 7 days
    DB->>ENG: faithfulness by query type, date, chunk source
    ENG->>ENG: identify prompt change as root cause
    ENG->>CI: revert prompt constraint, re-run eval
    RAGAS->>CI: faithfulness=0.88 — pass
    CI->>ENG: PR unblocked

What breaks

Ragas faithfulness has a ceiling problem

A generator that hedges everything — "The documents suggest...", "It may be the case that..." — achieves high faithfulness because vague claims are easy to support. But the answers are low-information. Faithfulness above ~0.9 in a system where users want direct, specific answers deserves scrutiny: you may have overconstrained the generator and traded hallucination risk for usefulness. Check answer relevance alongside faithfulness.

Context precision doesn't tell you which chunk caused harm

A context precision score of 0.5 means half your retrieved chunks were irrelevant. But Ragas gives you the aggregate score, not a per-query breakdown identifying the bad chunks. To diagnose which chunk types cause precision failures, you need to instrument chunk metadata (source, section, creation date) and join it against the Ragas per-row output. Ragas returns row-level scores in the result dataset — use them, don't average them away.

Reference-answer quality bottlenecks context recall

Context recall is only as good as the reference answers used to compute it. If your reference answers are incomplete — they mention three of the five facts that a complete answer requires — then context recall will overestimate retrieval performance. Building and reviewing reference answers for your golden dataset is an investment with a compounding return. The golden datasets article covers the three-source mix (handcrafted, de-identified production, synthetic expansion) that prevents coverage gaps.

LLM judge latency adds non-trivial overhead

Scoring a single RAG response across all four metrics takes 10–14 serial LLM calls if you run them naively. At 2–4s per call (for non-streaming judge calls), that's 20–56 seconds per evaluated example — far too slow to run synchronously in the production path. Options: batch offline asynchronously after the fact, run only the cheapest one or two metrics in the hot path, or use a faster judge model (GPT-4o mini at sub-second latency versus GPT-4o at 3–10s). evaluate() itself is a synchronous call, but Ragas parallelises the judge calls internally — RunConfig(max_workers=...) is the lever:

from ragas import evaluate
from ragas.run_config import RunConfig

# Parallelise the judge calls
results = evaluate(
    dataset,
    metrics=[faithfulness, context_precision],
    run_config=RunConfig(max_workers=8, timeout=60),
)

Judge noise makes hard thresholds flake

LLM judges are stochastic, and a 50-example golden set carries real sampling error — on the order of ±0.05 for a proportion metric near 0.75, before judge nondeterminism adds more. A hard assert score >= 0.75 against a baseline of 0.78 will pass one run and fail the next with zero code change, and a flaky quality gate gets deleted within a month, which defeats the whole point. Three mitigations, in ascending cost: run judge calls at temperature 0 (removes some variance, though providers do not guarantee determinism), set thresholds with margin — measure your baseline over 5–10 repeated runs and gate at the mean minus two standard deviations — or fail only when the median of three runs is below threshold. And size your expectations to the dataset: 50 examples reliably detects a ~0.1 swing in faithfulness; it cannot distinguish 0.76 from 0.78. If you need to catch 0.02 regressions, the answer is hundreds of examples, not a tighter threshold.

Score drift from judge model updates

Your Ragas scores are as stable as the judge model. If your judge model silently updates behavior — OpenAI and Anthropic both issue silent minor-version updates — faithfulness thresholds calibrated in January may no longer apply in July. Pin your judge model to a specific version where the API supports it, and re-calibrate thresholds against a small human-labeled set when upgrading. The same advice applies to any LLM-as-judge setup.

Multi-turn conversations need multi-turn metrics

Ragas's core four metrics assume a single-turn query–context–answer triplet. For conversational RAG where each turn builds on previous context, the conversation history changes what counts as a relevant chunk and what counts as a faithful response. Ragas v0.2 added multi-turn metrics; if you're evaluating a chatbot rather than a single-shot question-answering system, make sure you're using the multi-turn variants, or you'll undercount retrieval failures that stem from context accumulation rather than individual query failures.

Ragas in the broader eval stack

Ragas is not a complete eval system — it is a specialized tool for RAG-specific metrics. It tells you what is failing in your retrieval and generation, but not why at the system level or how often in production without additional instrumentation.

The full picture:

Layer	What it catches	Tool
Unit evals on golden dataset	Regressions in prompt changes, config changes	Ragas + pytest + CI
Ragas per-metric scoring	Retrieval vs generation failure localization	Ragas
Production sampling	Drift from model updates, new input distributions	Ragas batch scoring + Langfuse/Arize
Human spot-checks	Ragas judge calibration, edge cases	Manual review on low-score traces
LLM-as-judge for open-ended tasks	General response quality beyond RAG-specific metrics	Separate judge pipeline

Ragas context precision and recall don't fully replace a dedicated retrieval evaluation using labeled query–chunk relevance pairs — if your retriever evaluation needs to be comprehensive, annotating 1,000 query–chunk pairs for binary relevance is the gold standard. But for teams moving from zero retrieval measurement to something actionable in an afternoon, Ragas is the right starting point.

The prompt versioning and CI/CD article covers how to wire quality gates — including Ragas thresholds — into your pull request workflow so that regressions don't reach production.

The decision in practice

Start with faithfulness and answer relevance. They're cheap, reference-free, and immediately actionable. Run them on a 50-example golden dataset in CI from day one.

Add context precision when you have evidence of retriever noise — chunk metadata logging showing that certain source types frequently land in retrieved context but rarely contribute to the final answer is a signal. Budget for reference answers at the same time: the default precision judge needs them (or accept the reference-free variant's weaker semantics). Add it before you invest in a reranker, because it will tell you how much headroom exists.

Add context recall only when you have a maintained reference-answer corpus. If you're building that corpus anyway for golden datasets or human evaluation, the incremental cost of including it in Ragas runs is low.

Score production traffic from the start, not as an afterthought. A daily 5–10% sample costs a few cents to under a dollar for most applications (depending on traffic volume), and it catches the failure class — model provider updates, input distribution shift, new document types added to your corpus — that CI golden datasets structurally cannot catch. That class of failure is how polished RAG demos turn into unreliable production systems three months after launch.