Production Observability, Dashboards, and Quality Alerting

Knowing that something is wrong because a user filed a support ticket is the worst possible form of production monitoring. By the time a user articulates that the AI feature is behaving incorrectly, the regression has been affecting users for hours or days, you have accumulated a backlog of poor outputs, and you are starting a root cause analysis without any quantitative data about when the regression started, what changed, or how severe it is.

Production observability for AI features exists to change that: to detect quality regressions before users notice, to give engineers the data they need to diagnose the cause quickly, and to shorten the time from detection to resolution. This lesson covers what to instrument, how to store and visualise the signals, and how to build alerting that catches regressions early.

The AI Feature Instrumentation Checklist

Instrumentation for LLM features extends beyond standard API observability. In addition to the metrics you would log for any external API call, an LLM feature requires logging that captures quality, cost, and model-specific signals that are invisible to standard infrastructure monitoring.

Quality signals. Log an automated quality score for each LLM call where feasible. For high-traffic features, scoring every call with an LLM judge is expensive — use a sampling approach (score 10% of calls) or a fast automated scorer (NLI-based faithfulness, schema compliance check, output length check) that runs on every call. Quality signals should include: overall quality score (0–1 or 1–5 scale), per-dimension scores if your feature tracks multiple quality dimensions, and a binary flag for quality threshold violations (calls where the score falls below the minimum acceptable level).

Latency signals. Log time-to-first-token (TTFT), total generation time, and end-to-end pipeline latency as separate fields, not just a single response time. For RAG pipelines, log latency per step: embedding generation, vector retrieval, reranking, and LLM generation. Step-level latency logging is the difference between "the feature is slow" and "the vector retrieval step is slow under high load."

Cost signals. Log input token count, output token count, model name and version, and the computed cost per call (input tokens × input rate + output tokens × output rate). Aggregate cost per session, per user segment, and per feature variant. Without per-call cost logging, cost anomalies become visible only in the monthly billing statement — by which time the bill has already arrived.

Error signals. Log every error with sufficient context to diagnose the cause without re-running the failing call. For LLM errors: HTTP status code, provider error type (rate limit, timeout, invalid request, model overloaded), prompt token count at the time of error (long prompts correlate with timeouts), retry count, and whether the error was recovered by retry or escalated to a fallback. For application-level errors: schema validation failure with the raw output (stored securely with appropriate data handling), missing required fields, and any business logic violations in the output.

Model version and variant signals. Log the model version used for every call, the feature flag variant in effect, and the prompt version. Without these fields, correlating a quality regression with a specific model version or prompt change requires inference rather than direct evidence.

User segment signals. Log user segment, session ID, and the input category (if your feature has distinct input types that affect quality). These fields enable per-segment and per-category quality analysis that aggregate metrics cannot provide.

LangSmith Production Tracing Setup

LangSmith is the observability and evaluation platform that integrates most tightly with LangChain-based systems but works equally well with custom LLM integrations via its SDK. Production tracing in LangSmith captures the full call chain — prompt, model call, output, and any intermediate steps — and makes it queryable and filterable in the web UI.

SDK instrumentation. For a Python application, the LangSmith client wraps LLM calls with tracing automatically when LANGCHAIN_TRACING_V2=true is set and a LANGSMITH_API_KEY is configured. For non-LangChain integrations, use the @traceable decorator or the RunTree API to create traces manually:

from langsmith import traceable

@traceable(name="support-ticket-classifier", run_type="llm")
def classify_ticket(ticket_text: str) -> dict:
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_text}
        ]
    )
    return parse_and_validate(response)

Session grouping and metadata. Tag every trace with the session ID, user segment, feature flag variant, and prompt version. LangSmith allows filtering and grouping traces by any tag field, which is what enables questions like "show me all traces from the enterprise segment where the new prompt variant was active in the last 24 hours."

Quality score logging. Log quality scores as feedback on traces using the LangSmith feedback API. After scoring a call (via LLM judge or automated scorer), call client.create_feedback(run_id, key="accuracy", score=score_value). Feedback is then aggregable across any trace filter in the UI, and you can build dashboards that show quality score trends over time, segmented by any logged attribute.

Cost attribution. LangSmith automatically extracts token usage from provider responses and computes estimated cost per trace. Set up cost attribution by configuring the model-cost mapping in your LangSmith project settings. This enables per-project, per-session, and per-user-segment cost reporting without building a separate billing reconciliation pipeline.

Dashboard construction. Build a production quality dashboard in LangSmith by creating a saved filter set for your production project and pinning the quality score trend chart, error rate trend, and p95 latency trend to a shared dashboard view. LangSmith's dashboards are read-only for non-admin users by default, which makes them safe to share with product stakeholders without granting write access to the platform.

Braintrust Production Monitoring

Braintrust takes a slightly different architecture approach from LangSmith: it is built around experiments as the primary primitive, and production monitoring is implemented as a continuous experiment that logs and scores every production call.

Online evals with scoring functions. In Braintrust, a production scoring pipeline is configured as an online eval: a scoring function runs against logged production traces on a schedule (hourly or real-time via webhook) and posts scores back to the experiment. This means you can define the same scoring functions that run in CI and apply them continuously to production traffic without building a separate monitoring pipeline.

Score over time charts. Braintrust's experiment comparison UI shows score trends across experiments. For production monitoring, each daily or weekly rollup of production scores is an experiment data point, and the chart shows quality trend over calendar time. This makes it easy to see quality regressions as downward trend lines, and to correlate them with deployment events (also visible as vertical markers on the timeline if you log deployments to Braintrust metadata).

Experiment tracking. Every prompt change, model upgrade, and configuration change can be tagged as a new experiment in Braintrust. Production monitoring traces that occur after a change are automatically grouped under the new experiment, which means that quality comparisons before and after a change are built into the data model rather than requiring custom analysis.

Langfuse Open-Source Alternative

Langfuse is an open-source observability platform for LLM applications that supports self-hosting. For teams with data residency requirements — particularly in healthcare, finance, or public sector contexts — Langfuse provides the same core capabilities as LangSmith and Braintrust (tracing, quality score logging, dashboards, cost attribution) without sending production data to a third-party SaaS platform.

Self-hosted setup. Langfuse runs as a Docker Compose application or a Kubernetes deployment. The self-hosted version uses a PostgreSQL database for trace storage, which means trace data stays within your own infrastructure and security perimeter. Setup takes approximately four hours for an engineer familiar with Docker and Postgres.

Feature comparison for data-sensitive use cases. The primary trade-off with Langfuse versus SaaS alternatives is operational responsibility: you own the infrastructure, the backups, the upgrades, and the scaling. The primary benefit is data residency: production LLM traces, including any sensitive content they contain, never leave your infrastructure. For most teams, SaaS observability platforms are the right choice; for teams in regulated industries with data residency requirements, Langfuse is the practical alternative.

Building Quality Dashboards for Mixed Audiences

A single dashboard view cannot serve engineering, product, and cost audiences simultaneously. Engineers need raw signals and error details; product stakeholders need user-impact-level quality trends; finance and leadership need cost trends and efficiency metrics. Build separate dashboard views for each audience.

Engineering quality dashboard. Metrics: quality score (mean and p10) by hour and day, quality threshold violation rate, error rate by error type (rate limit, timeout, schema validation failure, other), TTFT and total latency (p50, p95, p99), and model version and prompt version currently active. Purpose: detect regressions, diagnose error sources, and verify that deployment changes have the expected effect. Audience: on-call engineers and AI feature owners.

Product quality dashboard. Metrics: weekly quality score trend (smoothed), user-facing error rate (calls where the fallback was shown rather than the AI output), task completion rate if measurable, and quality by user segment. Purpose: give product owners visibility into whether the AI feature is meeting its quality commitments and which user segments experience lower quality. Audience: product managers, engineering leads, and customer success teams.

Cost and efficiency dashboard. Metrics: daily and monthly cost trend, cost per session, token usage by category (input vs. output, per feature), model cost breakdown by version, and projected cost at forecast traffic levels. Purpose: identify cost anomalies, support budget planning, and evaluate the economic impact of model or prompt changes. Audience: engineering leads, finance, and leadership.

Dashboard maintenance. Dashboards that are not maintained become distrusted. Assign ownership to each dashboard — someone who is responsible for keeping the metrics accurate as the system evolves, removing stale metrics, and adding new metrics when new monitoring requirements emerge. A dashboard with five accurate, current metrics is more valuable than one with fifteen metrics, three of which are broken and two of which are stale.

Quality Alerting Strategy

Effective quality alerting catches regressions before users notice them without producing alert fatigue from false positives. Achieving this balance requires choosing the right alerting model for each signal type.

Static threshold alerts. A static threshold alert fires when a metric crosses an absolute value: "alert if quality score drops below 3.5" or "alert if error rate exceeds 5%." Static thresholds are appropriate for metrics with well-established acceptable ranges that do not vary significantly with traffic volume or time of day. They are easy to understand and implement but can produce false positives during expected low-traffic periods when smaller sample sizes cause metric variance.

Rolling baseline alerts. A rolling baseline alert fires when a metric diverges from its recent average by more than a defined amount: "alert if today's quality score is more than 0.3 points below the 7-day rolling average." Rolling baselines automatically adjust to expected trends and seasonal patterns. They are more appropriate for metrics that vary predictably (lower quality scores during periods of unusual input distribution, higher error rates during peak traffic) than static thresholds.

Anomaly detection alerts. Statistical anomaly detection identifies deviations that are statistically unlikely given the historical pattern of a metric, without requiring a manually specified threshold. Anomaly detection is appropriate for metrics where you have a rich historical baseline but are not sure what specific threshold to set. The trade-off is interpretability: an anomaly detection alert says "this is unusual" without specifying a specific threshold that was violated, which can make it harder to evaluate severity quickly.

Alert routing and severity. Not all quality alerts require the same urgency. Define severity levels and route accordingly. A quality score that drops 0.4 points below the rolling baseline warrants a Slack notification to the on-call engineer. A quality score that drops below the minimum acceptable threshold and triggers the fallback for more than 10% of calls warrants a PagerDuty alert. An error rate that spikes above 20% for more than 5 consecutive minutes warrants a P1 incident. Documenting the severity definitions and routing rules prevents ambiguity during incidents.

Alert fatigue prevention. The most dangerous alert is the one that fires so often that engineers learn to ignore it. Before enabling a new alert, back-test it against 30 days of historical metrics to verify that it would have fired on real incidents without firing more than twice per week on routine variance. Reduce alert sensitivity (raise the threshold, extend the rolling window, add a duration requirement) until the back-test false positive rate is acceptable.

MTTR for AI Quality Incidents

Mean time to resolution (MTTR) for an AI quality incident has a predictable structure: detection, diagnosis, fix, and verification. Each phase has an expected timeline, and the instrumentation choices you make in advance determine how long each phase takes.

Detection. With good alerting, detection time for a quality regression should be under 30 minutes from onset. Without production quality scoring and alerting, detection time is measured in days (when users start complaining). The investment in production observability pays off entirely in detection time reduction.

Diagnosis. Given production quality score trends correlated with deployment event logs, model version logs, and per-category quality breakdowns, a skilled engineer can diagnose most AI quality incidents in under 2 hours. The diagnostic questions are: when did the regression start? What changed around that time? Which categories or segments are most affected? Is this a quality score decline (output quality), an error rate increase (technical failure), or a cost anomaly?

Fix. For prompt regression causes: roll back the prompt to the last known-good version (minutes, if rollback infrastructure is in place). For model version regression causes: pin to the previous model version or apply a compensatory prompt update (hours to a day). For retrieval failures in RAG systems: investigate retrieval parameters, reranking configuration, or knowledge base update issues (hours to a day).

Verification. After applying a fix, run the production quality monitoring for 30 to 60 minutes before closing the incident, to verify that scores have returned to the expected baseline. Do not close an AI quality incident based on "the fix looks right" — verify with metrics before declaring resolution.

Post-incident review. Every significant AI quality incident warrants a structured post-incident review. The review should capture: timeline of events (when did the regression start, when was it detected, when was it diagnosed, when was it fixed), root cause, contributing factors (gaps in alerting, missing instrumentation, insufficient monitoring of the affected signal), and specific action items to prevent the same incident from occurring again. For AI quality incidents, the most common action items are: add a new alert on a signal that was visible but not monitored, add new examples to the eval set covering the affected input category, and improve the rollback procedure for the change type that caused the incident.