Tracing and observability tooling: LangSmith, Langfuse, Braintrust, and Arize compared

Your LLM feature shipped. Three weeks later, a senior engineer notices something in Slack: the chatbot is giving confidently wrong answers to a class of query it handled fine at launch. No errors. No latency spikes. Your uptime dashboard is pristine.

You have no idea when this started.

That gap — between what infrastructure monitoring sees and what's actually happening to your output quality — is the problem that LLM observability tooling exists to close. But "LLM observability" has become a category with four serious competitors, each making a different bet about what the actual workflow is and where the real value sits. Making the wrong choice means either missing the feedback signals that would catch regressions, or running a tool with enough friction that the team stops using it within a month.

This article is a concrete comparison of the four tools that show up most frequently in production AI stacks in 2026: LangSmith, Langfuse, Braintrust, and Arize Phoenix. The focus is on what each one actually does well, where each one has real costs or limitations, and how to wire any of them into a feedback loop that makes your eval infrastructure self-improving over time.

What "LLM observability" actually means

Standard observability — metrics, logs, distributed traces — is insufficient for LLM applications. The signal you care about is not "did the request succeed?" but "was the answer correct, faithful to the retrieved context, on policy, and free of hallucination?" None of those dimensions appear in an HTTP 200.

LLM observability tooling captures three layers:

Trace collection: every LLM call with its full input (prompt), output (completion), model name, token counts (input/output/cached), latency (time to first token, total), and a cost estimate. For agents, this extends to a nested span hierarchy capturing each reasoning step, tool call with its arguments and results, and sub-agent invocation.

Scoring: attaching a quality signal to each trace. This can be a binary human label, a Likert-scale human judgment, an automated LLM-as-judge score on specific rubric dimensions (faithfulness, relevance, groundedness), or a rule-based assertion (did the output match a regex, stay within a token budget, parse as valid JSON). See LLM-as-Judge: Calibrating Your Automated Evaluator for how to build scoring rubrics that are actually trustworthy.

Feedback loop: the mechanism by which scored production traces become regression test cases, and regression test cases become CI gates on prompt and model changes. This is where the tools diverge most sharply.

flowchart TD
    TC[Trace collection\nfull prompt + completion + metadata] --> SC[Scoring layer\nLLM-as-judge / human / rule-based]
    SC --> DS[Dataset management\ncurate, version, de-identify]
    DS --> CI[CI regression\nprompt change → golden dataset → gate]
    CI --> PM[Production monitoring\ndrift detection, sampling, alerts]
    PM --> TC

    style TC fill:#0e7490,color:#fff
    style SC fill:#a855f7,color:#fff
    style DS fill:#15803d,color:#fff
    style CI fill:#00e5ff,color:#0a0a0f
    style PM fill:#ffaa00,color:#0a0a0f

A tool that only does trace collection is an expensive logger. The value comes from closing the loop. The four tools span the full spectrum from "primarily a tracer" to "full eval lifecycle platform."

LangSmith

LangSmith is LangChain's hosted observability platform, and its greatest strength is also its most significant limitation: it's deeply integrated with the LangChain ecosystem.

If your stack is LangGraph or LangChain, LangSmith adds tracing in one line:

from langchain_openai import ChatOpenAI
from langchain_core.tracers.context import tracing_v2_enabled
import os

os.environ["LANGCHAIN_API_KEY"] = "..."
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_PROJECT"] = "prod-chatbot"

# Everything below is automatically traced — no other changes needed
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke("Explain KV cache eviction policies")

LangGraph agent traces come in as nested runs with tool call arguments, intermediate reasoning, and per-step latencies, all visualized in the LangSmith UI without any extra instrumentation. That developer experience is genuinely excellent.

For non-LangChain code, LangSmith provides the @traceable decorator and RunTree API, which work but require manual span construction — a meaningful extra surface area to maintain.

Where it fits well: teams where LangChain/LangGraph is the primary framework, who want zero-configuration trace collection and a clean UI for debugging agent trajectories.

Where it creates friction:

• Free tier: 5,000 traces per month. A single-user chatbot in active testing exhausts this. Production traffic at 500 req/day hits the ceiling in 10 days. Developer tier is priced around $39/month for higher limits (pricing as of mid-2026 — verify current plans).
• Self-hosting: not available on standard tiers; Enterprise only. For regulated industries or teams with data residency requirements, this is a hard blocker.
• Outside LangChain: the SDK works, but you lose the automatic span structure that makes agent traces readable. You're manually constructing a trace hierarchy that LangChain users get for free.
• CI gates: LangSmith added evaluation CI features in 2025 but they remain less mature than Braintrust's, particularly for automated PR-blocking based on regression scores.

LangSmith's dataset management is solid: you can add traces to datasets directly from the UI, version datasets, and run evaluations against them. The eval UI is good for exploration but the programmatic workflow for CI gating requires more wiring than Braintrust's.

Langfuse

Langfuse is open-source (MIT), self-hostable via Docker Compose or Kubernetes, and has first-class OpenTelemetry support — both emitting gen_ai.* spans and accepting OTLP traces from other instrumented code. That combination makes it the most architecturally flexible option.

The core SDK instruments calls with minimal intrusion:

from langfuse.openai import openai  # drop-in replacement for the openai client

# All calls are automatically traced
response = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this document: ..."}]
)

For frameworks Langfuse doesn't auto-instrument, the @observe() decorator nests spans automatically — each decorated function becomes an observation under its caller:

from langfuse.decorators import observe

@observe()
def retrieval(query: str) -> list[str]:
    return retrieve(query)

@observe()
def generation(query: str, chunks: list[str]) -> str:
    return generate(query, chunks)

@observe()
def rag_pipeline(query: str) -> str:
    chunks = retrieval(query)
    return generation(query, chunks)

Langfuse includes prompt versioning as a first-class feature: prompts are stored in Langfuse, versioned, and fetched at runtime, with each trace linked to the exact prompt version that produced it. This sounds minor but it's what lets you answer "did this regression start after the prompt change on Tuesday?" without digging through git history and guessing which deployed version was running at what time.

LLM-as-judge scoring in production is built in: you configure scoring rubrics (faithfulness, relevance, custom criteria) and Langfuse runs them against sampled traces asynchronously, attaching scores that appear in the trace UI and can be aggregated over time to detect drift.

For Ragas integration, Langfuse traces work directly as the input to RAG Evaluation with Ragas: the trace contains the query, the retrieved chunks, and the generated answer, which maps exactly to the four Ragas metrics.

Where it fits well: teams with data residency requirements or compliance constraints who need self-hosting; teams that want vendor-neutral OTel tracing they can redirect to multiple backends; teams building on frameworks other than LangChain.

The real cost of self-hosting: since v3, Langfuse needs a Postgres database, a ClickHouse instance for trace analytics, Redis, S3-compatible blob storage, and separate web and worker containers. On AWS that's roughly (illustrative, as of mid-2026):

t3.medium (web + worker):       ~$30/month
db.t3.micro (RDS Postgres):     ~$25/month
cache.t3.micro (ElastiCache):   ~$16/month
t3.medium + EBS (ClickHouse):   ~$40/month
S3 (traces + media):            ~$5/month
                                ───────────
Base:                           ~$116/month

Plus engineering time to maintain upgrades, backups, and the occasional incident at 2am. Langfuse Cloud avoids this but brings you back to per-observation pricing. The self-hosted path is genuinely viable for a team with infra maturity; it's not invisible overhead.

The Langfuse UI is functional rather than polished — it covers trace exploration, dataset management, and score visualization competently, but the UX for building CI pipelines requires more manual assembly than Braintrust's.

Braintrust

Braintrust's bet is that evaluation is a software engineering discipline, not a monitoring discipline. Its workflow is explicitly oriented around the loop: write evals, run them on datasets, gate CI on scores, promote good outputs to expand datasets, repeat.

The SDK syntax reflects this framing:

import braintrust
from autoevals import LLMClassifier  # Braintrust's scorer library

correctness = LLMClassifier(
    name="correctness",
    prompt_template="...",
    choice_scores={"correct": 1, "incorrect": 0},
)

# An experiment: run a dataset through a function and score the results
experiment = braintrust.init(
    project="customer-support",
    experiment="gpt-4o-mini-v2-prompt",
    dataset=braintrust.init_dataset(project="customer-support", name="golden-v3")
)

@braintrust.traced
def run_support_bot(question: str) -> str:
    # your actual implementation
    return call_llm(question)

for row in experiment.dataset:
    output = run_support_bot(row["input"])
    result = correctness(output=output, expected=row["expected"])
    experiment.log(
        input=row["input"],
        output=output,
        expected=row["expected"],
        scores={"correctness": result.score},  # scorers return a Score object
    )

print(experiment.summarize())

The experiment model is the key: every eval run is an experiment that records inputs, outputs, scores, and the exact model and prompt version used. You compare experiments against each other — "did moving from gpt-4o-mini to gpt-4o on this dataset improve correctness score by more than 5% while keeping cost per trace under $0.003?" — with a direct UI for the comparison, not a manual query against a database.

CI gating is Braintrust's strongest differentiator relative to the other three tools. You define evals in .eval.py (or .eval.ts) files, assert score thresholds in the file, and the braintrust eval runner exits non-zero when an experiment regresses below them. Wiring it into GitHub Actions:

- name: Run LLM eval suite
  run: braintrust eval support.eval.py
  # support.eval.py asserts correctness >= 0.85, faithfulness >= 0.90
  env:
    BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}

A failing threshold blocks the PR. This is the pattern Prompt Versioning and CI/CD for LLM Changes describes, and Braintrust is the tool that most directly implements it out of the box.

Where it fits well: teams doing serious iterative prompt engineering across multiple model options, where the question "did this change regress quality on our golden dataset?" needs a fast, automated answer on every PR.

Honest limitations:

• Self-hosting: no low-cost path. A hybrid deployment exists — the data plane (API and storage) runs in your own VPC while Braintrust hosts the control plane — but it's enterprise-tier. If you need data residency on a startup budget, that pushes you toward Langfuse or Phoenix.
• Pricing: Braintrust charges per scored trace, and LLM-as-judge scoring is not cheap. At 100K production traces/month with 5% sampling and judge scoring, you're looking at 5,000 scored traces. Depending on the judge model (GPT-4o runs at $2.50/1M input tokens as of mid-2026 — check current pricing), each judge call on a moderate-length trace costs roughly $0.002–$0.008. At 5,000 traces that's $10–$40/month just in judge costs, plus Braintrust's platform fee. Scale that to aggressive sampling rates and the bill is real.
• Production monitoring: Braintrust added production monitoring in 2025 but its primary workflow is still experiment-centric. For continuous production sampling and drift detection, Langfuse or Arize are more natural.

Arize Phoenix

Arize Phoenix is source-available (Elastic License 2.0 — free to run and self-host, not OSI open source; the OpenInference instrumentation packages are Apache 2.0), runs as a local or self-hosted service, and is OpenTelemetry-native. You install it as a Python package and it starts a local trace collector that any OTel-instrumented code can target.

import phoenix as px
from phoenix.otel import register

# Start the Phoenix collector (local by default)
px.launch_app()

# Register as the OTel trace provider — any OTel-instrumented SDK now sends here
tracer_provider = register(project_name="rag-pipeline")

# From this point, opentelemetry-instrumented LLM calls are traced automatically
from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is KV cache?"}]
)
# This call is traced automatically, including token counts and latency

The key differentiator is that Phoenix uses the OpenTelemetry GenAI semantic conventions throughout. Span attributes like gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.system are standard OTel attributes, not Phoenix-proprietary fields. That means your traces are portable: redirect the OTLP exporter endpoint from Phoenix to Langfuse, Jaeger, or any other OTel backend, and the data moves with you.

Phoenix has strong support for RAG evaluation via the phoenix.evals module, which provides the standard retrieval and generation metrics:

from phoenix.evals import (
    HallucinationEvaluator,
    RelevanceEvaluator,
    run_evals,
)
from phoenix.evals.models import OpenAIModel

# Pull traces from Phoenix and evaluate them
traces_df = px.Client().get_spans_dataframe(project_name="rag-pipeline")

hallucination_eval = HallucinationEvaluator(OpenAIModel(model="gpt-4o-mini"))
relevance_eval = RelevanceEvaluator(OpenAIModel(model="gpt-4o-mini"))

scores = run_evals(
    dataframe=traces_df,
    evaluators=[hallucination_eval, relevance_eval],
    provide_explanation=True,
)

Where it fits well: teams that want $0 tooling cost, own their infrastructure, and value being on the OTel standard. Also excellent as a local development tool — running px.launch_app() gives you a local trace UI that's far better than print() debugging.

Honest limitations:

• Managed cloud for production-scale scoring: Phoenix is self-hosted. Arize (the company) sells a managed Arize ML Observability platform that extends Phoenix into cloud-scale production monitoring, but that's enterprise pricing territory. For small-to-medium traffic, self-hosting is viable; for 10M+ traces/month, the operational picture gets complex.
• Dataset management and CI gates: Phoenix has a Datasets UI and an Experiments workflow added in 2025, but it's less polished than Braintrust's. The CI gate story requires more manual wiring: run evals via Python script, parse the output, fail the build on your own logic.
• UI polish: Phoenix's UI is functional and improving rapidly, but it does not match LangSmith's for agent trajectory visualization.

The OpenTelemetry foundation

All four tools now accept or emit OpenTelemetry gen_ai.* spans, and this matters more than which tool you pick. The GenAI semantic conventions define a standard trace shape for LLM calls:

gen_ai.system = "openai"
gen_ai.request.model = "gpt-4o-mini"
gen_ai.request.temperature = 0.7
gen_ai.request.max_tokens = 1024
gen_ai.usage.input_tokens = 842
gen_ai.usage.output_tokens = 156
gen_ai.usage.total_tokens = 998
gen_ai.response.id = "chatcmpl-..."
gen_ai.response.finish_reasons = ["stop"]

OTel v1.39 added gen_ai.tool.call.* attributes for MCP tool invocations, making agent trajectory tracing vendor-neutral for the first time. The practical implication: if you instrument using OTel's standard GenAITracer rather than a vendor SDK, switching backends is changing an environment variable:

from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Switch by changing this one URL:
exporter = OTLPSpanExporter(endpoint="https://your-langfuse-or-arize-or-any-otel-endpoint/v1/traces")

This is the bet Langfuse and Arize are making: that teams who invest in standard OTel instrumentation will be stickier than teams locked into proprietary SDKs. The counter-argument is that LangSmith and Braintrust's tighter integrations justify the lock-in with better out-of-the-box experience. Both arguments are correct, for different team situations.

Cost and scale math

The cost structure for LLM observability is often under-estimated. Here's a realistic back-of-the-envelope for a production application at 100K requests/month:

Traffic profile: 100,000 LLM calls/month
Avg tokens per call: 800 input + 200 output = 1,000 tokens/call
Total token volume: 100M tokens/month

TRACING COSTS:
  LangSmith Developer tier: ~$39/month (100K traces included)
  Langfuse Cloud: ~$49/month (200K observations; each call = ~1-3 obs)
  Braintrust: ~$0 for tracing; charges on scored traces
  Arize Phoenix self-hosted: ~$50-80/month infra

SCORING COSTS (5% sampling = 5,000 traces scored):
  GPT-4o-mini as judge (@$0.15/1M input tokens, ~2K tokens/judge call):
    5,000 × 2,000 tokens × $0.15/1M = $1.50/month (negligible)
  GPT-4o as judge (@$2.50/1M input tokens):
    5,000 × 2,000 tokens × $2.50/1M = $25/month
  
TOTAL REALISTIC RANGE: $40-130/month at 100K calls/month
  (dominated by platform tier, not judge model costs at 5% sampling)

At 10× scale (1M calls/month):
  Platform costs scale roughly 5-10× → $200-500/month
  Scoring at 2% sampling: still manageable if judge is GPT-4o-mini

The inflection point where costs become uncomfortable is roughly 1M calls/month with aggressive sampling and expensive judge models. Below that, any of the four tools is affordable. Above it, self-hosting Langfuse or Arize Phoenix to eliminate per-trace platform fees becomes economically significant.

What breaks in production

Trace volume surprises

A single LangGraph agent run with 10 tool calls and 3 LLM hops produces 13+ spans. At 10K agent runs/month, that's 130K+ spans — but 10K traces, and LangSmith's free tier counts traces: the 5K allowance burns down in about two weeks. On platforms that meter observations instead (Langfuse's tiers, for example), the same traffic counts as 130K+ billable units. The mismatch between "traces" (what one pricing page meters) and "observations" or "spans" (what another meters, and what one agent run actually generates) trips up teams during procurement.

Prompt version confusion

Without linking a trace to the exact prompt version that produced it, you cannot answer "did this regression start before or after Tuesday's prompt change?" Post-incident forensics without prompt version tracking is archaeological guesswork. Langfuse's prompt management solves this explicitly. LangSmith links to LangChain commit history if you use its built-in prompt hub. Braintrust's experiment model captures the prompt inline. Arize requires you to surface the prompt version as a span attribute manually.

LLM-as-judge drift

Your judge model is not static. If you're using a hosted model as judge and the provider silently updates it (which all of them do — see Offline vs Online Evaluation), your scoring rubric's behavior can shift without your scoring configuration changing. A judge that was calibrated at Pearson 0.82 against your domain experts can drift to 0.65 after a provider update. Lock your judge model to a specific version where the API supports it. Log the judge model name and version as a span attribute on every scoring call.

Feedback loop breaks

The most common failure mode: teams set up tracing, look at traces occasionally, but never build the pipeline from scored traces → golden dataset → CI gate. Observability without the feedback loop is expensive logging. The specific break point varies by tool: LangSmith users often have great trace coverage but manual dataset curation workflows. Braintrust users close the loop well on experiments but can neglect production sampling. Arize Phoenix users have the infra but need to build the scoring pipeline themselves.

Data in, data out

Every trace you send to a hosted LLM observability platform contains your users' prompts and your application's outputs. If your prompts contain PII, customer data, or proprietary information, you're shipping that to a third party. All four tools offer PII masking/scrubbing features, but the defaults send everything. Build your PII scrubbing layer at the trace instrumentation point, not at the platform — by the time data reaches the exporter, it's already left your application memory. See PII Handling and Privacy Controls.

Wiring the feedback loop: practical example

Here is what the closed feedback loop looks like with Langfuse, which offers the clearest separation between the components:

from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context
import random

lf = Langfuse()

@observe()
def answer_query(query: str) -> str:
    # Your actual LLM pipeline here
    result = run_pipeline(query)
    
    # Tag 3% of traces for judge scoring (tags live on the trace, not the span)
    if random.random() < 0.03:
        langfuse_context.update_current_trace(
            tags=["sample-for-scoring"]
        )
    
    return result

# Async scoring job (runs separately, not on the hot path)
def score_sampled_traces():
    traces = lf.fetch_traces(tags=["sample-for-scoring"], limit=100)
    
    for trace in traces.data:
        faithfulness_score = run_faithfulness_judge(
            input=trace.input,
            output=trace.output,
            context=trace.metadata.get("retrieved_chunks")
        )
        
        lf.score(
            trace_id=trace.id,
            name="faithfulness",
            value=faithfulness_score,
            comment="Auto-scored by judge v2.1"
        )
        
        # If score below threshold, add to golden dataset for review
        if faithfulness_score < 0.7:
            lf.create_dataset_item(
                dataset_name="faithfulness-failures-v3",
                input=trace.input,
                expected_output=None,  # Human to label
                metadata={"trace_id": trace.id, "auto_score": faithfulness_score}
            )

The scoring job runs on a schedule (cron, Celery, whatever), not on the request hot path. Low-scoring traces go into a dataset item queue. A human curator reviews them weekly, labels the expected output, and the dataset grows. CI runs the dataset on every prompt PR. The loop is closed.

The decision in practice

Pick your tool by answering three questions in order:

1. Are you on LangChain/LangGraph? If yes, LangSmith is the path of least resistance. The automatic trace collection for LangGraph agent trajectories is genuinely excellent and requires zero configuration. Buy a Developer tier seat before you hit the free limit.

2. Do you have a data residency or self-hosting requirement? If yes, Langfuse self-hosted or Arize Phoenix. Langfuse if you want a more complete platform including prompt versioning and a scoring UI. Arize Phoenix if you're heavily invested in the OpenTelemetry ecosystem or want to minimize managed dependencies. (Braintrust's hybrid data-plane deployment also satisfies residency, but only at enterprise pricing.)

3. Is evaluation a first-class engineering workflow — experiments, CI gates, regression tracking? If yes, Braintrust. The experiment-centric workflow and CI gate primitives are meaningfully better than the other three for teams doing serious iterative quality engineering. The trade-off is cloud-only and higher cost at scale.

For teams that don't fit cleanly into any of these: Langfuse cloud for the first six months. It covers all three layers (tracing, scoring, datasets) with a reasonable free tier, works with any framework, and the self-hosted option is available if you later need it. The LLM Observability: Monitoring Quality, Cost, and Drift article covers what dashboards and alerts to build on top of whichever backend you choose.

One thing that doesn't change regardless of tool: the feedback loop only works if someone owns it. The engineering decision is not just which platform to run, but which team is responsible for adding scored traces to the golden dataset, reviewing quality trends weekly, and blocking the build when regression scores drop. The tools make it possible. The process makes it happen.