AI Testing and Reliability Engineering Capstone Exercise

This capstone brings together every discipline covered in this course into a single, coherent engineering design. You are the lead AI reliability engineer for a company that has just shipped a major AI feature. Your job is to define the complete testing, evaluation, and reliability strategy that will keep that feature operating with confidence at scale.

This is a self-assessed design exercise. There are no right answers — only more and less rigorous ones. The self-assessment criteria for each deliverable describe what a complete, production-grade response looks like. Compare your response against those criteria honestly.

The System: AI-Powered Support Ticket Routing and Response Drafting

What it does. An enterprise SaaS company receives approximately 10,000 customer support tickets per day. The AI system processes each incoming ticket in two steps. First, a routing step classifies the ticket into one of eight categories (billing, account access, feature request, data export, integration issue, performance, security concern, other) and assigns a priority (critical, high, normal, low). Second, a response drafting step generates a suggested first-response draft for the support agent assigned to the ticket. The agent reviews, edits, and sends the draft. The draft is not sent automatically.

Technical architecture. Each incoming ticket triggers a pipeline: the ticket text is embedded and compared against a knowledge base of 12,000 past resolved tickets and 800 documentation articles (a RAG step). The retrieved context and ticket text are passed to the routing model and the drafting model in parallel (two separate LLM calls per ticket). The routing call uses a smaller, faster model (gpt-4o-mini or Claude Haiku) optimised for classification accuracy at low latency. The drafting call uses a larger model (gpt-4o or Claude Sonnet) optimised for response quality. Total pipeline budget is 6 seconds end-to-end.

Failure modes that matter. A mis-routed critical ticket that is assigned low priority can result in an SLA breach and enterprise customer churn. A response draft that contains fabricated information about the company's product capabilities can be sent by an inattentive agent, producing incorrect customer commitments. A pipeline that is slow (more than 8 seconds) delays the ticket appearing in the support queue, degrading agent response times during peak hours. A draft that consistently fails schema validation causes the fallback "no draft available" message to be shown to agents, forcing manual drafting for all tickets.

Volume and SLA context. 10,000 tickets per day = approximately 420 per hour average, with a peak of 900 per hour during business hours (9am–12pm in the primary customer timezone). The routing classification SLA is 30 seconds from ticket receipt to queue assignment. The draft should be available within 6 seconds of the routing step completing, so agents do not wait before opening a ticket. The routing model is called for every ticket; the drafting model is also called for every ticket by default, but agents may opt out of draft generation per ticket.

---

Deliverable 1: Evaluation Framework Design

Design the complete evaluation framework that will be used to measure quality, catch regressions, and validate future changes to this system before they reach production.

Your deliverable must specify:

1a. Metrics selection. For each of the two pipeline steps (routing and drafting), specify the primary quality metrics you will measure, the measurement method (automated rule-based, semantic similarity, NLI-based, LLM-as-judge, human rating), and why each metric is appropriate for that step. You should have at minimum two metrics per step. For the routing step, address both classification accuracy and priority accuracy — they have different cost profiles. For the drafting step, address at minimum faithfulness (to both the knowledge base and the company's actual product capabilities), response relevance, and tone appropriateness.

1b. Golden dataset specification. Specify the evaluation dataset for both pipeline steps. For each, address: total size, how examples are distributed across the eight ticket categories, what edge cases must be explicitly represented (minimum 15–20% of the set), how ground truth labels are produced (who labels, what qualifications, what decision procedure for ambiguous cases), and how inter-annotator agreement will be validated before the dataset is accepted as the quality standard. Address the specific challenge of labelling priority assignments, where the "correct" priority is often a business judgment call rather than an objective fact.

1c. LLM-as-judge design for response drafting. Design a complete LLM-as-judge rubric for evaluating the response drafts on the faithfulness dimension. Specify: the exact prompt structure for the judge (criteria definition, what each score level means with anchor examples at 1, 3, and 5), how you will mitigate positional bias (if applicable to your rubric design), which judge model you will use and why, and the calibration procedure you will follow before deploying the judge at scale. The calibration procedure must specify how many human-rated examples you need, what correlation threshold you require, and what you will do if the judge does not pass calibration.

Self-assessment criteria for Deliverable 1:

• Routing metrics are binary/multi-class classification metrics (precision, recall, F1 per class), not continuous scores — open-ended generation metrics applied to a classification step are a red flag.
• The priority labelling challenge is addressed with a decision procedure, not vague guidance.
• The golden dataset has explicit edge case categories: short tickets (under 20 words), tickets that span multiple categories, tickets in languages other than English, tickets with internal jargon or product model numbers, and tickets from account types with unusual entitlements.
• The LLM-as-judge rubric defines what "supported by context" means concretely, with anchor examples — not just a numeric scale with vague descriptors.
• Calibration procedure includes a minimum sample size of 50–80 human-rated examples, a specific Pearson r threshold (0.70 or higher), and a concrete action if calibration fails (revise rubric, change judge model, or add few-shot examples).

---

Deliverable 2: CI/CD Integration Plan

Design the CI/CD integration that will catch quality regressions in the routing and drafting components before any change reaches production.

Your deliverable must specify:

2a. Gate design. Specify what changes trigger a quality gate, whether the gate is blocking or non-blocking for each change type, and what quality metric thresholds must be met for the gate to pass. Change types to address: changes to the routing system prompt, changes to the drafting system prompt, changes to the retrieval configuration (embedding model, chunk size, top-k), changes to the reranking strategy, and model version pinning changes (switching from one pinned version to another). For each, state whether a blocking gate is justified and why.

2b. Cost management. At 10,000 tickets per day and a CI eval on every pull request, eval costs can become significant. Specify your eval dataset sampling strategy for PR-triggered evals versus scheduled evals, the fast-tier and slow-tier eval split (what runs on every PR versus what runs on a schedule or at merge), and the specific cost management techniques you will use (prompt caching, model version pinning in evals, parallel request batching). Calculate an approximate cost per CI eval run using the dataset sizes you specify in Deliverable 1, to verify your cost management approach is economically viable.

2c. Rollout strategy for a drafting prompt change. Walk through the complete rollout procedure for a high-confidence drafting prompt change that has passed offline evaluation. Specify: the shadow deployment configuration (duration, traffic percentage duplicated, what is measured, and what threshold must be met to proceed), the canary release stages (traffic percentages, observation period durations, promotion criteria at each stage, and the specific rollback triggers — one automatic, two manual — that apply at each stage), and how the previous prompt version is kept available for emergency rollback after full rollout.

Self-assessment criteria for Deliverable 2:

• The gate design distinguishes between routing and drafting changes — they affect different components and should have different quality metrics in their gates.
• The cost calculation is actually performed, not hand-waved. If your Deliverable 1 specifies a 300-example eval set with LLM-as-judge scoring, the cost per CI run should be calculable.
• The shadow deployment specifies a minimum duration (at least 72 hours for a system with weekday/weekend traffic variation) and a minimum call volume before shadow analysis is trustworthy.
• Rollback triggers are specific metric thresholds, not vague conditions. "Latency spikes" is not a rollback trigger. "p95 latency exceeds 10 seconds for any 15-minute window during the canary period" is a rollback trigger.

---

Deliverable 3: Load Test Plan

Design the load test that will validate whether the pipeline can handle peak production traffic within the 6-second end-to-end latency budget.

Your deliverable must specify:

3a. Test scenarios. Define at minimum three test scenarios: a sustained average load scenario (420 tickets per hour, sustained over 30 minutes), a peak load scenario (900 tickets per hour, sustained over 15 minutes), and a burst scenario (200 tickets arriving within a 2-minute window, simulating a sudden batch upload). For each scenario, specify the expected p50, p95, and p99 end-to-end latency targets and the maximum acceptable error rate.

3b. Tool choice and configuration. Choose k6 or Locust (both were covered in Lesson 6) and justify your choice for this system. The system has a multi-step pipeline (embedding, retrieval, two parallel LLM calls). Specify how your tool will: simulate realistic ticket text variance (not uniform repeated inputs), measure per-step latency breakdowns (not just total pipeline time), and handle the two parallel LLM calls (routing and drafting) which have different expected latencies. If you choose k6, sketch the relevant portions of the test script; if you choose Locust, sketch the task set.

3c. Acceptance criteria. Specify the numeric acceptance criteria that must be met for the feature to be considered load-test-approved for production launch. Address: end-to-end latency (p50, p95, p99 at peak load), per-step latency allocation (how you will distribute the 6-second budget across embedding, retrieval, routing model call, and drafting model call), maximum error rate at peak load, cost burn rate at peak load (tokens per minute and estimated hourly cost), and the specific provider rate limits (RPM and TPM) you must verify are sufficient for peak load.

Self-assessment criteria for Deliverable 3:

• The latency budget is allocated per step and the allocations sum to the 6-second total budget.
• The burst scenario tests a realistic failure mode — batch uploads are a common source of production traffic spikes for ticket systems.
• Provider rate limits are calculated for peak load and verified to have headroom. At 900 tickets per hour with two LLM calls per ticket and approximately 3,000 input tokens per call, the TPM requirement at peak is calculable.
• Acceptance criteria include both latency and cost — a plan that specifies latency criteria without cost criteria is incomplete.

---

Deliverable 4: Production Observability Setup

Design the production observability system that will detect quality regressions before users notice and enable fast incident diagnosis when issues occur.

Your deliverable must specify:

4a. Instrumentation checklist. List every field that will be logged on each pipeline call, with the logging rationale for each field. The list should be specific enough that an engineer could implement the logging without further design discussion. For multi-step pipelines, specify whether each field is logged per step (embedding, retrieval, routing call, drafting call) or at the pipeline level only.

4b. Dashboard specification. Design two dashboards: one for the engineering and on-call team, and one for the support operations lead. For each dashboard, specify the exact metrics displayed, the time ranges available, and the update frequency. The support operations lead does not need API error rates or p99 latency — they need to know whether the AI is helping or hurting their team's productivity. The engineering dashboard needs the raw quality and reliability signals.

4c. Alerting thresholds. Define a set of at least five alerts for this system. For each alert, specify: the metric being monitored, the alert type (static threshold, rolling baseline, or anomaly detection), the threshold value or baseline deviation, the minimum duration or sample size before the alert fires (to prevent false positives from single-point spikes), the severity level, and the routing destination (Slack channel, PagerDuty, or other). The alert set should cover: routing misclassification rate, drafting quality score, schema validation failure rate, end-to-end pipeline latency, and cost burn rate.

4d. Incident response playbook. Write a skeleton incident response playbook for a "drafting quality regression" incident: the AI drafts have suddenly become lower quality (vague, not personalised to the ticket context). The playbook should specify the first 30 minutes of the response: what metrics to check first, what deployment events to review, what diagnostic tests to run (oracle context test? per-category breakdown? model version log review?), and the decision tree for the most common root causes (model version change, prompt regression, retrieval quality degradation).

Self-assessment criteria for Deliverable 4:

• The instrumentation checklist distinguishes per-step logging from pipeline-level logging — logging everything at the pipeline level hides the step where a failure originates.
• The engineering dashboard and the support operations dashboard are genuinely different — not the same metrics with different labels.
• The five alerts cover at least three different signal types (quality, reliability, cost) — a set of five latency alerts is not a complete alerting strategy.
• The incident response playbook specifies concrete metric thresholds and lookup procedures, not vague activities. "Check the dashboards" is not a playbook step. "Check the 24-hour quality score trend on the engineering dashboard and compare to the deployment event log for the past 48 hours" is a playbook step.

---

Bringing It Together: What Makes a Complete Strategy

An AI reliability strategy is not the sum of four separate documents — it is a coherent system where each component reinforces the others.

Your evaluation framework feeds your CI gates: the same metrics and thresholds that define quality in your evaluation framework become the gates that block regressions in CI. Your CI eval results feed your production dashboards: the offline quality scores from CI provide the baseline against which production quality scores are compared. Your load test plan feeds your alerting thresholds: the latency numbers you measure during load testing are the basis for the p95 latency alerts in production. Your production observability feeds your evaluation framework: production traces where quality scores are low become candidates for the next iteration of your golden evaluation dataset.

When you review your four deliverables, check for this coherence. If your evaluation framework uses an accuracy metric that does not appear in your CI gate thresholds, why not? If your load test specifies a 6-second p95 target but your production alerting threshold is 10 seconds, which is the real standard? If your production dashboards do not show the same quality dimensions that your eval framework measures, how will you know when production quality diverges from offline eval quality?