Testing Non-Deterministic Systems: The Eval Mindset

By the end of this lesson, you can look at a Claude- or GPT-4-powered feature that "sometimes" fails and know whether you are looking at a genuine regression or expected output variance — instead of re-running the same prompt five times and guessing.

When a software engineer encounters a bug, the natural instinct is to write a test that reproduces it, fix the code, and verify the test passes. This approach has served the industry for decades. It rests on a foundational assumption: given the same inputs, the system produces the same outputs. Fix the code, run the test, pass or fail, done.

LLMs break that assumption completely. Send the same prompt twice at any temperature above zero and you will get different outputs. Send it a hundred times and you will get a distribution of outputs — some excellent, some mediocre, and occasionally one that is wrong in a specific way. There is no "the output" for a given input. There is only a distribution of outputs, and your quality engineering practice must work with distributions rather than against them.

This lesson establishes the mental model required to reason about AI quality at an engineering level. Everything that follows in this course builds on it.

Why Unit Tests Fail for LLM Output

Consider a unit test for an LLM-based feature that summarises customer support tickets:

def test_summarises_billing_ticket():
    ticket = "I was charged twice for my subscription in March..."
    result = summarise_ticket(ticket)
    assert result == "Customer reports duplicate billing charge in March."

This test fails as a quality measure on every dimension. The model might produce "User billed twice in March — possible duplicate charge" and the assertion fails even though the output is correct. It might produce "Billing error: double charge reported for March subscription" and the assertion fails even though that output is arguably better than the expected string. The test is not measuring quality — it is measuring exact string identity against an arbitrary reference output that was itself generated by a judgment call at the time of writing.

More fundamentally: even if you accept the fragility of exact-string matching, the test says nothing about 99.9% of the inputs your system will encounter. The model's behaviour on one ticket tells you almost nothing about its behaviour on the next thousand tickets, because LLMs are not functions — they are statistical models whose output quality varies systematically across different types of inputs.

The structural problem. Traditional unit tests verify that a specific code path produces a specific output. LLM integrations do not have code paths in the relevant sense — the logic is encoded in billions of model weights that transform a token sequence into a probability distribution over next tokens. You cannot read that logic, you cannot trace through it, and you cannot assert that a specific transformation will occur. What you can do is measure, aggregate, and evaluate across a distribution of inputs and outputs.

The sampling problem. Even if you accept distributional measurement, a single test case is a sample of one. Whether that sample is representative of the overall quality distribution depends entirely on how you chose it. If you write test cases for the easy, canonical inputs — because those are the ones where you know what correct looks like — you are measuring your best-case performance, not your production performance. Production inputs are messier, more ambiguous, and more diverse than test cases written from memory.

The oracle problem. To assert that an output is correct, you need an oracle — a reliable way to determine what the correct output actually is. For classification tasks with a fixed taxonomy, the oracle is the taxonomy. For generation tasks such as summaries, explanations, and recommendations, there is no single correct output. There are better outputs and worse outputs, and determining which is which requires judgment rather than comparison to a reference string.

What Correctness Means for AI Systems

Because there is no single correct output for most LLM tasks, quality measurement requires replacing the binary correct/incorrect with a correctness spectrum.

The correctness spectrum. Instead of pass/fail, AI quality measurement works with scores on relevant dimensions, aggregated across a representative sample of inputs. A summarisation system might be evaluated on accuracy (does the summary contain only claims supported by the source ticket?), completeness (does it capture all important information?), conciseness (is it appropriately short?), and tone (does it match the required professional register?). Each dimension is scored on a scale — commonly 1 to 5 or 0 to 1 — and aggregated across the evaluation set.

The key shift is from "did this specific call succeed?" to "what is our quality distribution across representative inputs?" A system where 95% of calls score 4/5 or higher on accuracy is a well-performing system, even though individual calls vary. A system where 70% of calls score 4/5 on accuracy but 15% score 1/5 has a tail problem that requires separate investigation.

Accepting variance as a feature, not a bug. Output variance is not something to eliminate — in most generation tasks, some variance in phrasing and structure is desirable. What you are trying to eliminate is variance in quality. A distribution of outputs that all meet your quality threshold, with natural variance in phrasing, is the target state. A distribution with high variance in quality — some excellent, some unacceptable — is the problem state.

The Five Dimensions of LLM Output Quality

Different LLM tasks prioritise different quality dimensions. Understanding which dimensions matter for your specific task is a prerequisite for designing a useful eval.

Accuracy. For factual tasks, does the output contain accurate information? Is every claim in the output supported by the source material (for RAG systems) or by the model's training knowledge (for general knowledge tasks)? Accuracy failures are the most consequential: an output that is fluent, well-structured, and confidently wrong causes real harm.

Faithfulness. Distinct from accuracy in RAG systems. Faithfulness asks whether the output contains only information present in the retrieved context, without introducing claims from the model's parametric knowledge. An output can be factually accurate but unfaithful if it introduces true information that was not in the retrieved documents — because the user has no way to know the output is drawing on knowledge beyond the cited sources.

Relevance. Does the output address what was actually asked? LLMs can produce highly accurate, faithful, well-written outputs that answer a slightly different question than the one asked. Relevance measures how well the output addresses the actual query or task.

Safety. Does the output avoid harmful, offensive, biased, or policy-violating content? Safety requirements vary dramatically by domain — a general-purpose chatbot has different safety requirements than an AI assistant for mental health support or financial advice. Define your safety requirements explicitly; do not assume they are self-evident.

Latency and cost. These are operational quality dimensions that are frequently omitted from eval frameworks but belong in any complete quality picture. An output that meets all quality thresholds but takes 12 seconds to generate, or costs $0.80 per call at production volume, may be functionally unusable. Latency and cost are constraints that shape which model, prompt design, and architectural choices are viable.

The Shift from Pass/Fail to Scoring and Distributions

The practical mechanism for measuring AI quality is the eval (evaluation run). An eval is the process of running your LLM system across a curated set of representative inputs, scoring each output on your chosen quality dimensions, and aggregating the scores into a quality distribution.

The structure of an eval. An eval has three components: an evaluation set (the inputs), a scoring function (the measurement), and an aggregation (the summary). The evaluation set is a curated collection of representative inputs, including edge cases and known-difficult examples. The scoring function can be an automated metric, a human rater, or an LLM judge (covered in detail in Lesson 2). The aggregation produces the distribution summary: mean score, score distribution, failure rate (percentage of calls scoring below a threshold), and performance by input category.

Quality thresholds. Once you have a baseline quality distribution, you define a quality threshold: the minimum acceptable score distribution that the system must achieve to be considered production-ready or regression-free. A typical threshold might be: mean accuracy score of 4.0/5.0 or higher, with no more than 5% of outputs scoring below 3.0/5.0, across the full evaluation set. Thresholds should be derived from what is actually acceptable for users, not from what is achievable with the current system.

Tracking over time. The value of a repeatable eval framework compounds over time. Once you have a baseline quality distribution and a threshold, every future change — prompt update, model upgrade, architectural change — can be evaluated against that baseline. Regressions become detectable before deployment rather than after users encounter them.

The Eval Mindset vs. The Test Mindset

The distinction between these two mindsets is not just about tooling — it reflects a fundamentally different relationship with uncertainty.

The test mindset assumes determinism and seeks binary correctness. It says: given these inputs, this code should produce this output. If it does not, something is wrong and must be fixed. The test mindset is comfortable with clear pass/fail gates and uncomfortable with probability.

The eval mindset accepts probabilism and seeks quality distributions. It says: given this distribution of inputs, our system should produce outputs that meet these quality thresholds on these dimensions. If the distribution degrades, we investigate. The eval mindset is comfortable with "95% of outputs meet the threshold" and is not destabilised by the fact that 5% do not — as long as the 5% is characterised and acceptable.

Both mindsets apply in AI systems, but to different layers. The test mindset applies to the deterministic code around the LLM call: the prompt builder, the response parser, the schema validator, the error handler, the retry logic. These are regular functions that can be unit tested with mocked LLM responses. The eval mindset applies to the LLM output itself: the quality of what the model produces across the distribution of real inputs.

The Testing Pyramid Redrawn for AI Systems

The traditional testing pyramid has unit tests at the base (many, fast, cheap), integration tests in the middle (fewer, slower), and end-to-end tests at the top (fewest, slowest). For AI systems, the pyramid gains a fourth layer and the middle layers change character.

Layer 1: Unit tests (deterministic code). These are unchanged from traditional software. Test prompt builders, parsers, validators, error handlers, and retry logic using mocked LLM responses. These tests are fast, cheap, and fully deterministic. They catch programming errors in the application code around the LLM, not quality issues in the LLM output.

Layer 2: Schema and contract tests. For LLM integrations that require structured output, test that the output schema is enforced correctly and that the integration handles schema violations as designed. These can be run against the live API with a small sample of test inputs. They test the integration contract, not output quality.

Layer 3: Eval suite (quality distribution). This is the new middle layer. An eval suite runs your system across a curated evaluation set of 50 to 500 inputs, scores outputs on quality dimensions, and compares the resulting distribution to baseline. Eval suites run in CI on pull requests that change prompts or model versions, and on a scheduled basis to detect silent model regressions. They take minutes to hours depending on set size and scoring approach.

Layer 4: Production observability (continuous monitoring). The top of the pyramid is continuous quality monitoring in production: instrumenting live calls to track quality signals, latency, cost, error rate, and model version over time, with alerting on degradation. This is covered in depth in Lesson 9.