AI Product Requirements and Testing

Standard QA practices break down when applied to AI features without modification. A unit test that asserts on a specific output string will be flaky. A story that says "the AI correctly classifies the support ticket" without defining what "correctly" means at the boundary case level is not a specification. A regression test suite that checks deterministic logic cannot detect when a model update has degraded your AI feature's output quality.

Testing AI features requires different approaches, not just more of the same. This lesson covers what changes and what stays the same.

Why Standard Acceptance Criteria Break for AI Features

Standard acceptance criteria for a deterministic feature look like this: given input X, when action Y occurs, then output Z is produced. This is precise, testable, and binary. Either Z is produced or it is not.

AI feature outputs are probabilistic. The same input can produce different outputs across calls. The boundary between correct and incorrect output is often a matter of degree, not a binary. The failure mode is not "wrong output every time" but "output that falls below an acceptable quality threshold on some percentage of inputs."

AI-aware acceptance criteria need to account for this. Instead of "the AI produces the correct classification," the criterion should be "the AI produces the correct classification for at least 90% of inputs in the evaluation set, with recall on the high-priority category above 85%." Instead of "the AI response is accurate," the criterion should be "the AI response is rated as accurate or mostly accurate by the evaluation set judges on at least 88% of test cases."

This shift from binary pass/fail to threshold-based criteria applies across the entire AI feature testing surface.

Writing AI-Aware User Stories

The standard "given/when/then" user story format adapts to AI features with specific modifications.

Given the setup: include the context that shapes the AI output (the user's account type, the document being processed, the history of prior interactions).

When the action: specify the trigger precisely (the user submits the form, the background job runs, the ticket arrives).

Then the outcome: express the expected output as a range or threshold, not a single correct value. "Then the AI suggests a priority category that matches the ground truth category or an adjacent category (one step higher or lower) in at least 90% of cases" is an AI-aware then clause.

Add a when the AI is uncertain or when the input is ambiguous clause that specifies the expected fallback behaviour. "When the ticket content is too ambiguous for the classifier to produce a confidence score above 0.7, then the ticket is routed to the manual review queue" is the edge case clause that standard user story format would omit but AI features require.

Evaluation Sets: Build Them Before the Feature

An evaluation set (eval set) is a labelled dataset of inputs with known correct or acceptable outputs, used to measure AI feature performance. Every AI feature needs one before it ships. Building it after the feature is built is both harder (the feature's current performance anchors your judgement of what the correct label should be) and less useful (you cannot measure whether the feature meets the acceptance criterion because you have no pre-defined measurement instrument).

A minimal eval set for a classification feature: 50 to 100 examples from your actual data, with labels applied by subject matter experts rather than the AI being evaluated. The examples should include representative cases from each category, edge cases near category boundaries, and a small number of adversarial examples (inputs that are designed to confuse the classifier).

A minimal eval set for a generation feature (summarization, drafting, extraction): 20 to 40 examples with human-written reference outputs or human quality ratings. For generation tasks, exact match against a reference output is rarely the right metric. More useful metrics: rating on a quality rubric (is the output accurate, complete, and concise?), ROUGE or BLEU scores for summarization tasks, or LLM-as-judge scoring.

The LLM-as-Judge Pattern

For generation tasks where output quality cannot be objectively measured by exact match or simple metrics, a second LLM can be used to evaluate the quality of the first LLM's output. This is the LLM-as-judge pattern.

How it works. A judge prompt presents the original task, the original input, and the candidate output to a judge model (often a more capable model than the one being evaluated) and asks it to score the output on defined quality dimensions. For a summarization task: accuracy (does the summary contain only information from the source?), completeness (does it cover the main points?), and conciseness (is it appropriately brief?). Each dimension is rated on a scale with clear rubric descriptions for each score level.

Benefits. LLM-as-judge scales to thousands of examples without human labelling cost. It can evaluate nuanced quality dimensions that exact-match metrics miss. For rapid iteration on prompt changes, LLM-as-judge provides fast feedback on quality changes.

Risks. LLM judges have systematic biases: they prefer longer responses, may favour outputs from the same model family as the judge, and can be manipulated by flattering language in the evaluated output. Judge scores are not ground truth. They correlate with human judgement at the population level but diverge on individual examples. Use LLM-as-judge for relative comparisons (did prompt version B outperform prompt version A?) rather than as an absolute quality standard. Maintain a human-labelled golden set for calibration.

Regression Testing for AI Features

When a model provider updates a model version, your AI feature's output quality may change without any change on your end. This is a class of production risk that conventional software does not face.

To detect model regression, you need: an eval set with stable reference outputs, an automated eval run that can be triggered on demand or on a schedule, and alerting when eval scores fall below defined thresholds.

Running evals on model version changes. Some providers announce model updates in advance. Others update silently. A weekly scheduled eval run against your production eval set catches regressions that a silent model update introduces. When a regression is detected, you can pin to a specific model version while investigating and fixing.

Detecting prompt regression. A prompt change that improves performance on the cases you tested may degrade performance on cases you did not test. Running the full eval set on every prompt change before shipping to production catches these regressions.

Human Evaluation and the Golden Set

Automated evals and LLM-as-judge cover most testing volume, but some evaluation requirements need human assessment. Human evaluation is appropriate when: the quality dimension is genuinely difficult to specify in a rubric, when LLM judge calibration is uncertain, and when the consequences of misclassification are significant enough to require human sign-off.

The golden set is a small, stable set of high-quality human-labelled examples maintained as the authoritative ground truth for the feature. It is used to calibrate automated eval scores (does LLM-as-judge agree with human raters on these specific examples?) and as the final gate for launch decisions and major changes.

Keep golden sets small (20 to 40 examples), stable (do not change them frequently), and diverse (cover representative cases, edge cases, and adversarial examples). Recalibrate the golden set periodically as the feature's use case evolves.

The Testing Pyramid for AI Systems

The testing pyramid for AI systems has three layers, each with a different purpose.

Unit tests cover the deterministic logic in your AI feature: the prompt construction, the response parsing and validation, the confidence threshold logic, the fallback routing, the error handling. These are standard unit tests and should behave as such. If a function takes a raw API response and converts it to your application schema, that function should be unit tested with mocked API responses.

Eval suites cover the AI output quality. They test whether the model produces acceptable outputs on the defined set of inputs. These are not unit tests. They do not have a binary pass/fail for individual examples; they measure aggregate performance against thresholds.

Integration tests cover the system behaviour end to end: does the feature correctly handle the full request lifecycle including API calls, database writes, downstream notifications? These should use mocked LLM responses for reliability and speed, with a small set of actual API call integration tests kept behind a flag for explicit testing against the live API.

Try It: AI-Graded Practice

The exercise above is self-assessed. The exercise below is graded automatically, so you can get direct feedback on whether your rewritten acceptance criterion actually expresses quality as a threshold across an evaluation set with a risk-based routing rule.

Try It: Run the Code

The exercise above is about writing an AI-aware acceptance criterion in prose. This exercise is different: it is deterministic, not AI-graded. Your code actually runs against fixed eval data and is checked against exact expected outputs — no judgment call involved.