A/B Testing LLM Variants

Running a controlled experiment to determine whether LLM variant A or B produces better outcomes is structurally more complex than running a typical web experiment. In a standard A/B test, each user sees a deterministic page or feature variant. In an LLM experiment, each user receives a non-deterministic output from a probabilistic model — and the outcome metric may itself be produced by another LLM judge, adding a second layer of probabilism to the analysis.

This lesson covers the experimental design, sample size calculation, statistical analysis, and common pitfall mitigation for A/B experiments on LLM systems.

Experimental Design for Non-Deterministic Systems

Good experimental design for LLM variants requires careful thought about what you are measuring, how you are measuring it, and what confounds might corrupt the measurement.

Define the experiment scope clearly. The experiment should test one variable at a time. If you change both the system prompt and the model version, you cannot attribute quality differences to either change independently. If you change the system prompt and also change the retrieval configuration, the experiment measures the combination, not either change in isolation. LLM experiments that test multiple simultaneous changes are almost always uninterpretable.

Define the treatment and control. The control is the current production system. The treatment is the system with the proposed change. The control must be running in production during the experiment period — "the offline eval baseline" is not a valid control for a live experiment because it was measured at a different point in time, on a curated dataset, and without the full production input distribution.

Blocking confounds. LLM output quality varies with input characteristics. If your traffic is not uniformly distributed across input types (and it never is — most systems have seasonal patterns, user segment differences, and query type distributions that vary across the day and week), random assignment does not guarantee that the treatment and control groups see the same input distribution. Verify that the treatment and control groups have similar distributions on key input characteristics (query length, query category, user segment) before relying on experimental results.

The unit of randomisation. For LLM experiments, randomise at the user or session level, not at the request level. Request-level randomisation means the same user might receive variant A for one request and variant B for the next, which creates a confounded user experience and makes it impossible to measure user-level satisfaction metrics. User-level randomisation ensures each user consistently experiences one variant.

Defining the Primary Metric

The primary metric is the number that determines the experiment outcome. It must be defined before the experiment starts, not chosen after looking at results.

Automated quality scores as primary metrics. For AI features where user behaviour does not provide a direct quality signal, an automated quality score (LLM-as-judge score, task completion rate measured programmatically, or a composite quality score) is the primary metric. The advantage is that it is measurable on every interaction without requiring user action. The disadvantage is that it measures proxy quality, not user-experienced quality — a higher judge score does not guarantee a better user experience.

Behavioural primary metrics. For AI features where user behaviour signals quality directly — user ratings, edit rates, copy rates, task completion rates, or return rates — behavioural metrics are preferable to proxy quality metrics because they measure actual user value. The disadvantage is that they require more interactions to accumulate sufficient signal: a rating from 5% of users is weaker evidence than an automated quality score on 100% of interactions.

Composite metrics. For complex AI features, a composite primary metric that combines multiple signals (60% judge score + 40% user engagement rate, for example) can capture quality more comprehensively than any single metric. Composite metrics are harder to interpret and more prone to gaming, but they reduce the risk of optimising one dimension at the expense of others.

Primary metric requirements. Whatever you choose as the primary metric, it must be: measurable on every interaction (or a large enough fraction to achieve statistical significance), validated as a leading indicator of actual user value (not just a metric that is easy to compute), and defined with a minimum detectable effect — the smallest difference between variants that would constitute a meaningful improvement worth shipping.

Sample Size Calculation for LLM Experiments

Sample size calculation for LLM experiments follows the same statistical principles as standard A/B tests, with two additional complications: LLM quality metrics are often noisier (higher variance) than typical web metrics, and the metric scale may not be binary.

For binary outcome metrics (task completed vs. not completed, user rated positively vs. negatively):

n = (Z_α + Z_β)² × p(1-p) / δ²

where Zα is the z-score for the significance level (1.96 for α=0.05), Zβ is the z-score for the desired power (0.84 for 80% power), p is the baseline rate (current variant's success rate), and δ is the minimum detectable effect (the smallest improvement worth detecting).

For a feature with a current task completion rate of 72% and a minimum detectable effect of 5 percentage points, you need approximately 850 interactions per variant (1,700 total) for 80% power at α=0.05.

For continuous quality score metrics (LLM-as-judge scores on a 1–5 scale):

n = 2 × (Z_α + Z_β)² × σ² / δ²

where σ² is the variance of the quality score in the baseline, and δ is the minimum detectable effect in score points.

For a feature with a baseline quality score mean of 3.8 and standard deviation of 0.9, and a minimum detectable effect of 0.2 points, you need approximately 640 interactions per variant (1,280 total).

The variance problem. LLM quality scores have higher variance than most web metrics because: the LLM output is non-deterministic (same input, different quality score on different calls), the LLM judge is itself non-deterministic, and user inputs vary widely in difficulty. Higher variance means larger required sample sizes. If your quality score has a standard deviation of 1.2 rather than 0.9, the required sample size increases by 78%.

Minimum sample heuristics. For most LLM features, minimum sample sizes for reliable experiment results are: 1,000 to 2,000 interactions per variant for continuous quality score metrics, 2,000 to 5,000 interactions per variant for behavioural binary metrics with typical base rates. Run fewer interactions and your experiment is likely to be underpowered — you will fail to detect a real improvement even when one exists.

Statistical Significance Testing for LLM Quality Scores

Once you have collected sufficient data, you need to test whether the observed quality difference between variants is statistically significant or within the range of natural sampling variance.

Frequentist approach. For continuous quality score metrics, a two-sample t-test tests whether the mean quality scores for the two variants are significantly different. For binary metrics, a chi-squared test or a proportion z-test is appropriate. The frequentist approach requires: a pre-defined significance level (alpha = 0.05 is the standard), a pre-defined minimum detectable effect (set before the experiment), and a commitment to not looking at results until the target sample size is reached (peeking inflates false positive rates).

The frequentist approach is appropriate when: you have a well-defined hypothesis before the experiment, you can commit to running the experiment to completion without early stopping, and you have a strong prior on the minimum effect size worth detecting.

Bayesian approach for small samples. LLM experiments frequently accumulate data more slowly than standard web experiments, because LLM features often serve fewer interactions per day. For experiments where reaching the frequentist sample size would take weeks, a Bayesian approach can provide actionable probability estimates from smaller samples.

A Bayesian A/B test models the quality score distributions of both variants as probability distributions and estimates the posterior probability that the treatment variant is better than the control. Rather than producing a binary significant/not-significant result, it produces a probability: "there is an 87% probability that Variant B is better than Variant A by at least 0.1 points." This probability can be used to make a decision with explicit uncertainty acknowledgment.

For LLM quality experiments with limited data, a Bayesian t-test using conjugate priors (a normal-normal model for continuous quality scores) is computationally tractable and produces interpretable posterior distributions. The key prior specification is the expected quality score range — typically easy to set from historical data.

Multi-armed bandit for prompt selection. If you are running ongoing prompt optimisation experiments across many prompt variants rather than a single A vs. B comparison, a multi-armed bandit (MAB) algorithm can allocate traffic adaptively to the better-performing variants as evidence accumulates. Thompson sampling is a Bayesian MAB algorithm that works well for quality score optimisation: it samples a quality estimate from the posterior for each variant and routes the request to the variant with the highest sampled estimate. Over time, traffic concentration shifts toward the better variant without requiring an explicit "experiment over" decision.

Guardrail Metrics

Every LLM experiment should define guardrail metrics alongside the primary metric. Guardrail metrics are metrics that must not degrade — the experiment is automatically stopped if any guardrail metric exceeds its threshold, regardless of primary metric performance.

Latency guardrails. The new variant must not increase p95 latency beyond an acceptable threshold. A variant that produces better quality but increases p95 latency from 3 seconds to 8 seconds may not be worth shipping, depending on the use case.

Cost guardrails. The new variant must not increase per-request cost beyond an acceptable percentage. An improvement of 0.2 quality score points that doubles per-request cost may not be economically justified.

Safety guardrails. The new variant must not produce a higher rate of safety policy violations (harmful content, PII exposure, off-topic responses) than the control. Safety guardrails should be monitored continuously during the experiment, not just at the conclusion.

Error rate guardrails. The new variant must not produce a higher rate of API errors, schema validation failures, or timeout events than the control. A variant that has better quality scores but a higher error rate is trading one problem for another.

Common Pitfalls in LLM A/B Experiments

Novelty effect. Users sometimes respond differently to a new feature variant simply because it is different, not because it is better. For LLM features, the novelty effect is less common than for visual redesigns, but it can appear when a prompt change produces noticeably different output style. To mitigate: run the experiment for at least one full week to allow the novelty effect to decay.

Segment imbalance. If your user base has distinct segments (enterprise vs. self-serve, experienced vs. new users, users who submit short vs. long inputs) and the random assignment produces unequal segment distributions between treatment and control, the quality difference you measure may reflect segment composition rather than variant quality. Stratify your random assignment by key segment dimensions, and report primary metric results separately by segment.

Metric gaming. A prompt that inflates LLM-as-judge scores by using verbose, authoritative-sounding language may produce higher judge scores than a more concise, equally accurate prompt, even though the concise prompt is preferable for users. Guard against metric gaming by maintaining a human-reviewed holdout sample evaluated by people, not just by the automated primary metric, during major experiments.