LLM Observability: Monitoring Quality, Cost, and Drift in Production

Your LLM-powered feature shipped on a Thursday. On Friday the product metrics looked fine — engagement was up, no error spikes, latency at P95 was 620ms. On the following Wednesday, a developer on your biggest enterprise account posted in the shared Slack channel: your docs assistant had been telling their team to call POST /v1/completions-sync — an endpoint you deprecated in January — complete with plausible example code and auth headers. Not once — consistently, across dozens of conversations, to paying customers. Nothing had crashed. Nothing had triggered an alert. The model had been wrong for six days and your monitoring stack had no idea.

This is the gap that LLM observability exists to close.

Why traditional APM is incomplete

Standard APM captures what every good observability stack should: request rates, error rates, latency percentiles, saturation. These metrics are necessary — you absolutely need to know when the model API is returning 429s or when your P99 latency crossed a SLA boundary. But they are blind to the question that actually matters: was the answer right?

A hallucinated response has a 200 status code. A response that confidently answers a different question than the user asked has a latency of 380ms. A response that surfaces PII from another user's document looks, at the HTTP layer, identical to a correct one. Infrastructure metrics cannot distinguish between them.

The measurement problem compounds in multi-step agent pipelines. A tool-calling agent might make five retrieval calls, call an external API, reason over the combined context, and produce a final answer — all as a single user-visible response. Traditional distributed tracing was built for microservices where each hop either succeeds or fails. An LLM hop can "succeed" at the transport layer while producing output that is subtly, silently wrong. You need tracing that captures what went into each LLM call, what came out, and whether the output was any good — not just whether the TCP connection completed.

The four signal types

Building on the reference architecture for a production LLM application, LLM observability adds four distinct signal categories to your standard infrastructure stack.

Output quality measures whether individual responses are faithful to the source documents, relevant to the user's question, and coherent. This is the signal most teams instrument last and need most urgently first.

Behavioral drift detects distribution-level shifts: average response length suddenly dropped 40%, refusal rate increased from 2% to 12%, the topic distribution of conversations shifted. These shifts often precede — or explain — quality degradation. A silent model version update by your provider shows up as drift before users start complaining. A bad prompt change shows up the same way.

Cost anomalies track per-request token consumption, aggregate cost by feature or workflow, and alert on spikes. An agent that started including much longer context for some query pattern will surface here before it appears on your monthly bill.

Safety violations capture triggered guardrails, PII detected in outputs, policy refusals, and injection attempts. These should fire immediately without sampling — every violation is signal.

Tracing: the foundation

Distributed tracing is where LLM observability starts. The goal is a trace for every LLM interaction that captures what was sent to the model, what came back, and the timing and token cost of every hop in between.

The standard format is OpenTelemetry (OTEL). Every major observability vendor — Datadog, Langfuse, Arize, Helicone — ingests OTEL spans. This matters because the LLM tooling market is moving fast; you do not want your instrumentation tied to a single vendor.

A minimal OTEL span for an LLM call carries:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import time

tracer = trace.get_tracer("llm-app")

def call_model_with_tracing(messages: list, feature: str, model: str = "gpt-4o") -> str:
    with tracer.start_as_current_span("llm.completion") as span:
        span.set_attribute("llm.feature", feature)
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.input_messages", str(messages))

        start = time.monotonic()
        response = client.chat.completions.create(
            model=model,
            messages=messages,
        )
        latency_ms = (time.monotonic() - start) * 1000

        span.set_attribute("llm.input_tokens", response.usage.prompt_tokens)
        span.set_attribute("llm.output_tokens", response.usage.completion_tokens)
        span.set_attribute("llm.latency_ms", latency_ms)
        span.set_attribute("llm.output", response.choices[0].message.content)
        span.set_status(Status(StatusCode.OK))

        return response.choices[0].message.content

Notice what that span just did: it wrote the full prompt and the full completion into your observability store. Do that naively and the trace store quietly becomes the largest PII repository in your company — every customer name, contract number, and pasted email your users ever sent, retained on whatever default your vendor ships. Decide the payload policy before the first span, not after the first audit: scrub known PII patterns before export (Datadog and Langfuse both ship input/output scrubbing for exactly this reason), or store payloads truncated or hashed and keep full text only for the sampled traces you actually score. Set a retention window measured in weeks rather than the default forever, and gate raw-payload access separately from metrics access. The drift pipeline described below doesn't need raw text anyway — it runs on embeddings.

Streaming changes what you measure

Most production chat and agent traffic is streamed, and the snippet above quietly assumes it isn't. Three things change. Latency splits in two: time to first token (TTFT) is what the user feels, while total generation time drives throughput and cost — record them as separate span attributes, because a regression in one hides in an average of both. Token usage only arrives when the stream completes — with the OpenAI SDK, pass stream_options={"include_usage": True} and read usage off the final chunk. And the output must be accumulated as it streams if you want it on the trace at all.

def call_model_streaming(messages: list, feature: str, model: str = "gpt-4o") -> str:
    with tracer.start_as_current_span("llm.completion") as span:
        span.set_attribute("llm.feature", feature)
        span.set_attribute("llm.model", model)

        start = time.monotonic()
        first_token_at = None
        chunks, usage = [], None

        stream = client.chat.completions.create(
            model=model,
            messages=messages,
            stream=True,
            stream_options={"include_usage": True},
        )
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                if first_token_at is None:
                    first_token_at = time.monotonic()
                chunks.append(chunk.choices[0].delta.content)
            if chunk.usage:  # present only on the final chunk
                usage = chunk.usage

        span.set_attribute("llm.ttft_ms", (first_token_at - start) * 1000)
        span.set_attribute("llm.total_ms", (time.monotonic() - start) * 1000)
        span.set_attribute("llm.input_tokens", usage.prompt_tokens)
        span.set_attribute("llm.output_tokens", usage.completion_tokens)
        return "".join(chunks)

Everything downstream — output capture, quality sampling, the judge queue — hooks the end of the stream, not the moment you start writing bytes to the user.

For a RAG pipeline, you want separate child spans for the retrieval step, the context assembly, and the model call — so you can see where latency and token cost are actually coming from. A 1.2-second response time reads very differently if 800ms is in retrieval versus 800ms in the model's prefill phase.

sequenceDiagram
    participant U as User
    participant APP as Application
    participant RET as Retriever
    participant LLM as Model API
    participant OBS as Observability

    U->>APP: query
    APP->>OBS: start trace (trace_id, feature_tag)
    APP->>RET: retrieve(query)
    RET-->>APP: chunks [span: 45ms, 4 chunks, 1,200 tokens]
    APP->>LLM: complete(context + query)
    LLM-->>APP: response [span: 820ms, 2,140 in / 310 out]
    APP->>OBS: end trace (quality_sample_flag: true/false)
    APP-->>U: response
    OBS-->>OBS: async: if sample → queue for scoring

If you are already using the AI gateway pattern, the gateway is the natural place to inject trace IDs and capture token counts — you get instrumentation without touching application code.

Quality scoring in production

Instrumentation gives you traces. Quality scoring tells you whether the answers in those traces were any good.

Embedding-based similarity

The cheapest quality signal that runs on every request is embedding cosine similarity between the model's response and the source documents it retrieved. If the model was supposed to answer from retrieved chunks and the response has a cosine similarity below ~0.65 with all of them, that is a hallucination candidate.

import numpy as np
from openai import OpenAI

client = OpenAI()

def faithfulness_score(response: str, source_chunks: list[str]) -> float:
    """Cheap proxy for hallucination: similarity between response and sources."""
    # One batched API call: response first, then the chunks
    embeddings = client.embeddings.create(
        input=[response] + source_chunks,
        model="text-embedding-3-small"
    ).data
    resp_embedding = embeddings[0].embedding

    similarities = [
        np.dot(resp_embedding, c.embedding) /
        (np.linalg.norm(resp_embedding) * np.linalg.norm(c.embedding))
        for c in embeddings[1:]
    ]
    return max(similarities)  # best match score

Against the hosted embeddings API this is a ~100–300ms round trip, so run it async off the response path like everything else; if you serve a small embedding model locally, the same check drops to 5–20ms and can sit on the critical path. Either way it costs a fraction of a cent per request. It catches responses that have drifted far from source material but misses cases where the model confidently produces plausible-sounding content that happens to embed near the source topic. It is a filter, not a complete solution.

LLM-as-judge faithfulness scoring

For the quality signal that actually matters, you need a judge model. The standard approach: sample 1–5% of traffic, send the (question, retrieved context, response) triple to a judge with a structured scoring prompt, and record the verdict.

import json

FAITHFULNESS_PROMPT = """You are evaluating whether an AI response is faithful to the provided source context.

Source context:
{context}

Question: {question}
AI Response: {response}

Score the response on a scale of 1-5 where:
1 = Contains claims not supported by or contradicting the context
3 = Mostly faithful but includes minor unsupported claims
5 = Completely faithful, every claim is directly supported by the context

Respond with JSON: {{"score": <1-5>, "reason": "<one sentence>"}}"""

def judge_faithfulness(question: str, context: str, response: str) -> dict:
    result = client.chat.completions.create(
        model="gpt-4o-mini",  # cheaper judge; gpt-4o for higher stakes
        messages=[{
            "role": "user",
            "content": FAITHFULNESS_PROMPT.format(
                context=context,
                question=question,
                response=response
            )
        }],
        response_format={"type": "json_object"},
        temperature=0.0,
    )
    return json.loads(result.choices[0].message.content)

GPT-4o-mini-class judges agree with human raters on faithfulness roughly 75–80% of the time at about 500× lower cost than human review. That agreement rate is enough to catch systematic regressions — a prompt change that drops average faithfulness from 4.2 to 3.1 will surface in your sampled scores within hours. It is not enough to rely on for individual high-stakes decisions, but production monitoring is a statistical problem, not a per-request audit.

{"type": "eval-pipeline", "title": "Where quality scoring fits in the safety net"}

Behavioral drift detection

Drift is the signal that something changed when you did not explicitly change it. Provider silent model updates are more common than vendors admit. The prompt versioning and CI/CD article covers gating intentional changes; drift detection catches unintentional ones.

Track these distributions daily, compared against a 7-day rolling baseline:

• Response length (mean + P90 token count). Sudden shortening often means the model started refusing or deflecting more. Sudden lengthening means it started hedging or adding caveats.
• Refusal rate. Track the fraction of responses containing refusal language (phrases like "I'm unable to", "I can't assist with"). A 2% to 12% jump with no prompt change means something in the model's safety tuning changed.
• Topic distribution. Embed a sample of responses and run cluster assignments against your topic taxonomy. A shift in cluster proportions catches when the model starts routing responses to different semantic spaces.
• Context window size (P50, P90 per workflow). This is both a cost signal and a quality signal. Context rot — the degradation in output quality that occurs as context size grows — is a documented, reproducible phenomenon. If your P90 context size grew from 12K to 22K tokens after a retrieval change, expect quality scores to follow downward.

flowchart LR
    subgraph "Daily drift check"
        SAMPLE["Response sample\n(1-5%)"] --> EMB["Embed responses"]
        EMB --> DIST["Compute distributions\n(length, topic, refusal rate)"]
        DIST --> COMPARE["Compare to 7-day baseline"]
        COMPARE -->|"delta > threshold"| ALERT["Alert: possible drift"]
        COMPARE -->|"within bounds"| OK["No action"]
    end

    subgraph "Context rot monitor"
        CTX["P90 context size\nper workflow"] --> CORR["Correlate with\nquality scores"]
        CORR -->|"degradation found"| TICKET["Flag context review"]
    end

    style ALERT fill:#ff2e88,color:#111
    style TICKET fill:#ffaa00,color:#0a0a0f

Cost monitoring and attribution

Token costs are non-linear with traffic in LLM systems in a way that infrastructure costs are not. An agent that starts including longer context for some query type can double its per-request cost without any change in request volume. A retrieval pipeline that fetches more chunks after a configuration change will amplify that into more prompt tokens on every downstream call.

The correct unit of cost attribution is not the provider account or even the model — it is the product feature or workflow. This requires tagging every LLM call at the SDK level with a feature identifier. The tag flows through the trace and gets aggregated in your observability store.

Monthly cost breakdown at 1M requests/day across three features:

Feature             Avg input tokens  Avg output tokens  Daily cost
─────────────────────────────────────────────────────────────────────
Customer support    2,400             280                $4,320
Document search     1,800             120                $2,070
Code assistant      3,600             850                $8,730

Total                                                    $15,120/day

"Code assistant" is 57% of cost despite ~30% of traffic.
Context size P90: customer support = 8K, code assistant = 18K.
→ Code assistant context is the investigation target.

Set up alerts on:

• Per-feature cost increasing more than 20% week-over-week without corresponding traffic increase
• Per-request token count exceeding 2× the baseline P90 for any workflow
• Output token ratio (output_tokens / input_tokens) jumping — often indicates the model started generating longer hedged responses

The token economics article covers the underlying cost mechanics; what observability adds is the attribution layer that tells you which workflow is responsible for which portion of the bill.

{"type": "token-cost", "title": "Input vs output vs cached: what a month of tokens costs"}

Safety monitoring

Safety signals should not be sampled — they should fire on every violation. What to capture:

• Guardrail triggers: how often your input or output guardrails are firing, segmented by guard type (PII, toxicity, off-topic, injection attempt). Trends here are as important as the absolute count — a rising injection attempt rate might mean a specific prompt injection attack is being tried at scale.
• PII in outputs: flag any output that contains patterns matching known PII formats (SSN, credit card, email). A rising rate after a retrieval change is a serious incident.
• Policy refusals: track the fraction of requests that hit a refusal, along with the topic clusters that cluster around high refusal rates. This helps distinguish legitimate refusals from over-triggering safety filters.

The input and output guardrails article covers the implementation of the guardrails themselves. Observability captures whether they are firing at the right rate.

What breaks

Sampling bias blinds you to tail failures

A 2% sample is statistically useful for catching systematic regressions in average quality. It is nearly useless for catching rare but severe failures — the 0.1% of responses that contain hallucinated medical advice, or the 0.05% that leak PII. Safety monitoring cannot be sample-based. Quality monitoring can, but you need to be honest that your 2% sample gives you a 98% chance of missing any single bad response.

The mitigation is a two-tier approach: full-traffic cheap signals (embedding similarity, guardrail triggers, context size) plus sampled expensive signals (LLM-as-judge faithfulness, relevance, and coherence scoring).

The eval dataset goes stale

You build quality score baselines at launch. Over six weeks, your real query distribution shifts — users found a new use pattern, you added a new feature, a seasonal topic became prominent. Your eval baselines no longer represent production traffic. The scores continue looking fine against the old baseline while production quality on the new query distribution is quietly degrading.

The fix is continuous baseline refreshing: weekly automated sampling of recent production queries to update the reference distribution, and quarterly golden dataset review to add coverage for new query patterns. The golden datasets article covers the curation mechanics.

Cost attribution is retrofitted too late

Teams typically add feature tags to LLM calls after the first monthly bill arrives and causes alarm. By then, multiple services have scattered direct provider calls with no consistent tagging scheme. Retrofitting attribution across a dozen services is weeks of work. Adding the tag at SDK initialization takes ten minutes before the first call.

The practical answer: define your feature taxonomy before writing the first LLM call, add a mandatory feature_tag parameter to your internal LLM client wrapper, and fail loudly if it is absent. Your future self will thank you when a cost spike arrives.

Judge model instability

The LLM-as-judge approach has a subtle failure mode: the judge model itself changes. If your quality scoring uses GPT-4o-mini and OpenAI silently updates that model, your quality score distributions shift — but that shift reflects the judge changing, not your product changing. You may trigger false drift alerts or, worse, miss real regressions because the new judge happens to score them higher.

Mitigation: version-pin your judge model where your provider supports it, run your judge on a fixed golden test set weekly, and alert if the judge's scores on those fixed examples shift by more than one standard deviation.

flowchart TD
    JUDGE["LLM-as-judge\n(gpt-4o-mini)"] -->|"weekly calibration"| GOLD["Fixed golden set\n(100 labeled examples)"]
    GOLD --> CAL["Judge score vs\nhuman label agreement"]
    CAL -->|"agreement < 70%"| WARN["Alert: judge calibration drift"]
    CAL -->|"agreement ≥ 70%"| OK["Judge is stable"]

    style WARN fill:#ff2e88,color:#111
    style JUDGE fill:#a855f7,color:#fff

Latency added by tracing

Synchronous quality scoring on the critical path is a latency disaster. An LLM-as-judge call takes 500ms–2s. Every quality scoring call must be asynchronous, queued after the response is sent to the user. The trace ID is emitted with the response; the quality score is attached to the trace record later. Your observability store must support late score attachment.

This sounds obvious until the first implementation where someone adds a synchronous scoring call to the response handler and wonders why P50 latency doubled.

Tooling landscape (mid-2026)

Tool	Best for	Key trade-off
Datadog LLM Observability	Teams already on Datadog; want APM + LLM in one pane	Best APM integration; weaker standalone eval tooling
Arize	Teams with ML background; strongest on drift and model quality	Steeper learning curve; most powerful for statistical analysis
LangSmith	LangChain-heavy stacks; agent chain tracing	Deep LangChain integration; somewhat tied to that ecosystem
Langfuse	GDPR-constrained deployments; self-hosted requirement	Open-source, self-hostable; community support rather than SLA
Braintrust	Eval-first teams; pairs well with A/B testing	Evaluation focus over operational monitoring
Helicone	Zero-code instrumentation; proxy-based	Proxy adds a hop; less flexible for custom metrics

All six ingest OTEL spans, which is the only vendor-neutral baseline you should require. If you're not on any of them yet, Langfuse is a reasonable first choice — you can self-host it in an hour, get traces flowing, and migrate to a managed tool later without losing your instrumentation code.

OpenLLMetry (by Traceloop) is worth mentioning separately: it is an OTEL instrumentation layer that auto-instruments LangChain, LlamaIndex, and direct OpenAI/Anthropic SDK calls without code changes. It feeds any OTEL-compatible backend. If you want zero-code trace collection that is not tied to a specific observability vendor, it is the most practical starting point.

The decision in practice

Observability stacks tend to grow in layers. The first layer is almost always tracing — get spans flowing for every LLM call, capture token counts and latency. That alone will show you where time and cost are going, which is valuable within the first week.

Cost attribution is next. Add the feature tag to every call before the stack grows — retrofitting it across a dozen services is weeks of work. The cost of skipping this step is measured in engineering-weeks later.

Quality monitoring is the third investment. Start with embedding-based similarity on full traffic — cheap, catches gross hallucinations, and runs async off the hosted API (or synchronously only if a locally served embedding model keeps it under ~20ms). Add LLM-as-judge on a 2% sample once you have baselines established. Drift detection comes fourth — once you have enough quality score history to establish a meaningful baseline, nightly distribution checks take an hour to set up and provide surprisingly early warning on real regressions.

Safety monitoring is not a layer — it is day one. Any output-level guardrail trigger should be logged immediately, without sampling.

The teams that get this right treat observability as part of the prompt versioning and CI/CD pipeline: every prompt change ships with updated quality score baselines, every model swap is accompanied by a drift monitor reset, and every incident post-mortem asks not just "what went wrong" but "why did our monitoring not catch this first."

That last question has a surprisingly consistent answer in early-stage LLM systems: the monitoring stack was measuring infrastructure and ignoring output quality. The HTTP 200s were all green.