Production Observability for LLM Systems — Building Production LLM Systems

What you'll learn

Define the minimum required trace data for each LLM call in a production system and explain why standard API logging is insufficient
Compare LangSmith, Braintrust, Helicone, and Langfuse on the dimensions that matter for a production engineering team
Design distributed tracing for a multi-step RAG pipeline that enables root cause isolation for quality failures
Build a quality score tracking dashboard and alerting system that detects regressions before users report them

Standard application observability — request logs, error rates, p95 latency — is necessary but insufficient for LLM systems. An LLM call can succeed technically (HTTP 200, valid JSON response) while failing qualitatively (wrong answer, unfaithful generation, hallucinated information). Traditional monitoring does not see qualitative failures at all.

LLM observability extends traditional observability to capture what happens inside the AI layer: not just whether the API returned successfully, but what prompt was sent, what response was received, how many tokens were used, what the latency breakdown was, whether the output passed quality evaluation, and what the sequence of steps was for multi-turn or multi-step pipelines.

Without this visibility, debugging a quality regression in production means analysing user complaints with no access to the specific prompts, responses, or retrieval context that caused them. With it, a quality regression is diagnosable within minutes.

What to Trace

Every LLM call in production should emit a trace that captures the following:

Prompt contents. The full prompt as sent to the API, including system prompt, retrieved documents, conversation history, and user message. Logging only the user message and the response loses the context needed to diagnose why the model responded the way it did.

Completion contents. The full model response, including any reasoning steps, tool call requests (for agent systems), and the final output before and after post-processing. If you parse and transform the raw output before returning it, log both the raw output and the parsed result.

Token counts. Input tokens, output tokens, cache hit tokens (if applicable), and total tokens per call. Token counts are the primary cost driver and the primary indicator of prompt bloat over time.

Latency breakdown. Time to first token (TTFT), total generation time, and where relevant, time spent in retrieval and post-processing. TTFT is the user experience metric for streaming applications; total generation time is the throughput metric for batch applications.

Model version. The exact model version used for each call. Providers update models without always surfacing it obviously in API responses. Logging model version enables correlating quality changes with specific model updates.

Cost per call. Computed from token counts and current pricing. Logging per-call cost enables cost attribution to features, users, and model versions without relying on the provider's billing dashboard (which has a multi-hour lag).

Quality scores. If you run automated evaluation on outputs (LLM-as-judge scores, RAGAS metrics), attach them to the trace. This enables quality tracking across the same infrastructure as technical metrics.

Session and run IDs. For multi-turn conversations and multi-step agent pipelines, group individual LLM calls under a session or run ID. This enables viewing the complete sequence of events for a specific user interaction or agent run.

Warning

Prompt contents can contain sensitive user data. Before enabling full prompt logging, determine whether your data retention and privacy policies permit storing user query text and any personal information that may appear in it. If they do not, log sanitised or tokenised versions, or apply a retention policy that deletes prompt content after a defined period. Observability tooling that stores prompts in a third-party service has additional data residency implications.

LLM Observability Platform Comparison

Four platforms dominate the LLM observability space as of mid-2026: LangSmith, Braintrust, Helicone, and Langfuse. They share core tracing capabilities but differ on integrations, evaluation features, and operational model.

LangSmith (from LangChain) provides deep integration with LangChain and LangGraph. If your agent system is built on LangGraph, LangSmith is effectively the native observability layer: graph execution traces, node-level timing, and state snapshots at each step are captured automatically. The evaluation and dataset management features are tightly integrated with the tracing data. The limitation is that LangSmith's value proposition is strongest if you are on the LangChain ecosystem; teams using raw tool use or other frameworks get tracing capabilities but lose the native integration advantages.

Braintrust positions itself as an evaluation-first observability platform. The core differentiator is its dataset, experiment, and scoring workflow: you can run evaluations directly in Braintrust, store evaluation results alongside traces, and compare experiment results across prompt versions. For teams that want a unified platform for both evaluation and production observability, Braintrust is a strong choice. The tracing SDK supports OpenAI, Anthropic, and custom LLM calls.

Helicone is a lightweight observability proxy: requests to LLM providers are routed through Helicone's proxy, which logs them without requiring SDK instrumentation. The zero-friction setup is the primary advantage — no code changes required beyond changing the API base URL. The tradeoff is that it captures only what passes through the proxy (prompt, response, token counts, latency) rather than application-level context that you would add via SDK instrumentation.

Langfuse is open-source and self-hostable. For teams with strict data residency requirements that rule out third-party SaaS, Langfuse running in your own infrastructure is the primary option among the major platforms. It supports manual SDK instrumentation, LangChain integration, and OpenAI/Anthropic SDK wrapping. The tracing model and evaluation features are comparable to commercial platforms, with the tradeoff of operational overhead.

The selection decision:

LangGraph/LangChain ecosystem: LangSmith
Evaluation-first workflow, commercial SaaS acceptable: Braintrust
Zero code change setup, commercial SaaS acceptable: Helicone
Self-hosted / data residency required: Langfuse

Distributed Tracing for Multi-Step Pipelines

A single user query in a RAG system triggers multiple operations: embedding the query, searching the vector index, (optionally) reranking results, constructing the prompt, calling the LLM, parsing the output, and returning the response. Each step has its own latency and can fail independently. A distributed trace captures all steps as a tree of spans, with timing and context attached to each span.

Trace structure for a RAG pipeline. The root span represents the full request. Child spans represent: embedding (with the query text and embedding model), retrieval (with the query, top-k results, and retrieval latency), reranking (with input candidates and output reranked candidates), prompt construction (with the assembled prompt token count), LLM call (with full prompt and response), and output parsing (with the raw model output and parsed result).

Attaching context to spans. Each span should carry: the step name, start and end timestamps, any inputs and outputs relevant to that step, and any errors. For retrieval spans, include the retrieved chunk IDs so you can correlate retrieval results with response quality. For LLM call spans, include token counts and model version.

Correlating traces to quality. The most powerful observability use case is correlating specific trace attributes with quality outcomes. Which retrieval configurations produce the most faithful answers? Which query types produce the longest LLM calls? Which document types appear in low-quality responses? These correlations require trace data that goes beyond simple request logs.

Cost Dashboards

A cost dashboard for an LLM system should answer four questions: what does each feature cost today? Is cost growing faster or slower than traffic? Which model versions are most cost-efficient? Where are the cost anomalies?

Per-feature cost attribution. Tag every LLM call with a feature identifier in the trace metadata. Aggregate daily token counts and costs by feature. This enables the "which features drive cost?" analysis that is essential for prioritising cost optimisation work.

Cost rate vs. traffic rate. Plot daily API cost and daily request count on the same chart. If cost is growing faster than requests, something is increasing per-request cost: prompt bloat, context growth, model tier changes, or retrieval returning more documents. If cost is growing at the same rate as requests, per-request cost is stable.

Model cost comparison. For teams running multiple model tiers, track cost per 1,000 requests broken down by model version. This reveals whether model routing decisions are working as intended and flags when a cheaper model has been inadvertently replaced with an expensive one.

Quality Score Tracking

Quality metrics should be tracked over time with the same discipline as technical metrics. A quality score that drops from 4.2 to 3.8 over two weeks is as significant as a p95 latency increase from 800ms to 1,200ms.

Automated quality scoring in production. Run LLM-as-judge evaluation on a random sample of production responses — 5 to 10% of traffic is typical. Attach the quality score to the trace. Aggregate daily average quality scores by feature and query category.

Quality-latency-cost dashboard. The most useful operational view is a combined dashboard that shows, for each LLM feature: average quality score, p95 latency, and average cost per request, all over time. This gives you the three-way tradeoff view needed to make informed model routing, caching, and prompt decisions.

Alerting on quality regression. Set quality score alerts at a threshold below your historical baseline. A drop of 5 percentage points over a 24-hour rolling window is a reasonable regression alert threshold. Alert destinations: Slack notification to the engineering team, PagerDuty for critical features (those directly in user-facing flows with quality SLAs). Pair the alert with a direct link to the observability platform filtered to the time window where the regression started.

Session replay for debugging. When a quality alert fires, the investigation starts with session replay: retrieving the traces for the affected time window and examining specific examples where quality scores are low. Good observability platforms make this a one-click operation from the alert to the trace. The trace shows exactly which step in the pipeline produced the low-quality result: a retrieval failure (wrong context), a prompt issue (model misinterpreting instructions), or a model regression (same prompt, lower quality output after a model update).

Catching a silent model regression through quality score monitoring

Platform Engineer

Context

A B2B SaaS company ran a RAG-powered contract clause extraction feature that their customers used to populate deal summaries. The feature had been stable for three months with a consistent LLM-as-judge quality score of 4.1 to 4.3 on their weekly evaluation sample. The team used Braintrust for observability and had configured a quality score alert below 3.7.

Action

On a Tuesday afternoon, the quality alert fired: the 24-hour rolling average quality score had dropped to 3.4. The team opened Braintrust and filtered traces to the preceding 6 hours. Session replay showed a consistent pattern: the extracted clause structures were correct in schema but the values were abbreviated — where a previous response might extract 'Net 30 days from invoice date', the new responses were extracting 'Net 30'. Sorting traces by model version in the metadata revealed that OpenAI had silently updated gpt-4o to a new version at 10:47 AM that day. The new version produced more concise outputs by default, which passed schema validation but reduced the completeness of extracted values. The team temporarily rolled back to the pinned model version (gpt-4o-2024-05-13) while updating their system prompt to explicitly request full, unabbreviated values.

Outcome

The rollback restored quality scores to 4.2 within 30 minutes of identifying the root cause — under 45 minutes from alert to resolution. Without the quality score monitoring and session replay, the regression would have been discovered when customers reported incomplete deal summaries, likely 24 to 48 hours later. The team added model version pinning to their standard deployment checklist and implemented a weekly model version check that alerted when the provider updated a pinned model.

Knowledge check

A production RAG system begins returning low-quality answers. The engineering team has LLM call traces but the traces only capture the user query and the final LLM response — not the retrieved documents or the assembled prompt. What can the team determine from the available traces, and what can they not determine?

Select one answer.

Quick check

What does a proxy-based observability tool give up in exchange for requiring no code changes to install?

Select one answer.

Exercise

Your Task

Design the observability architecture for the following production LLM system: a multi-step research assistant that, given a user question, (1) generates 3 search queries from the question, (2) retrieves 5 documents per query using a RAG pipeline, (3) deduplicates and reranks the retrieved documents, (4) generates a structured research summary, and (5) cites the specific document passages that support each claim in the summary. The system handles 500 queries per day and the team cares about: diagnosing which step fails when quality is poor, tracking cost per query and per step, detecting when the research summary contains claims not supported by the retrieved passages, and replaying specific failed sessions for debugging. Specify: what to trace at each step, which observability platform you would use and why, the quality metrics you would compute in production, and the alerting thresholds you would set.

Your reflection

Did you complete this exercise? What did you find? (Saved locally in your browser)

Key takeaways

Standard API logging (request, response, HTTP status) is insufficient for LLM systems. Every production LLM call requires tracing: full prompt contents, full completion, token counts by type, latency breakdown, model version, and per-call cost. Qualitative failures are invisible to technical metrics alone.
LangSmith is the natural observability choice for LangGraph/LangChain systems. Braintrust provides the strongest evaluation-tracing integration. Helicone requires zero code changes via proxy. Langfuse is the self-hosted option for data residency requirements.
Distributed tracing for multi-step pipelines requires a span tree that captures each pipeline stage independently — embedding, retrieval, reranking, prompt construction, LLM call, output parsing — with inputs, outputs, and timing per stage. Without stage-level spans, distinguishing retrieval failures from generation failures is impossible.
Quality score tracking must use the same temporal discipline as technical metrics: daily aggregates, week-over-week trend charts, and automated alerts at a threshold below the historical baseline. A 5-percentage-point drop over a 24-hour window is a reasonable regression alert threshold.
Session replay — the ability to examine the full trace of a specific failed user interaction — is the highest-value debugging capability for LLM systems. Select an observability platform that supports it natively or ensure your custom tracing infrastructure enables it.