Designing Evaluation Frameworks for LLM Systems

An eval framework is only as good as its inputs and its scoring. A poorly curated evaluation set gives you a number that feels like quality measurement but is actually measuring a biased sample of easy cases. A poorly designed scoring function gives you a number that correlates weakly with what users actually experience. Both problems are common, both are fixable, and both are expensive to discover late.

This lesson covers the engineering decisions that determine whether your eval framework produces reliable signals or misleading ones.

Golden Dataset Curation

The evaluation dataset is the foundation of your entire eval framework. Every quality number you produce is conditional on the quality and representativeness of this dataset. A dataset that is too small produces noisy estimates. A dataset that is not representative produces estimates that look good but do not predict production performance. A dataset labelled inconsistently produces a quality bar that shifts depending on who labelled which examples.

Dataset size. For a binary classification task, 200 to 500 labelled examples is typically sufficient to detect a 5-percentage-point change in accuracy with reasonable statistical confidence. For generation tasks where quality is scored on a multi-point scale, you need larger datasets to estimate the score distribution reliably — 300 to 1,000 examples is a reasonable range, with larger sets needed for tasks with more output variability. The key principle: your dataset should be large enough that a meaningful quality regression produces a statistically detectable change in your aggregate metrics. If a 10% increase in failure rate on a 20-example dataset produces a confidence interval that spans 40 percentage points, your dataset is too small to be informative.

Diversity requirements. Representative coverage is more important than raw size. An evaluation set of 200 examples that covers all important input categories is more valuable than 1,000 examples that are all drawn from the same input type. Before collecting examples, enumerate the input categories your feature will encounter in production: for a customer support classifier, that means all supported ticket categories, tickets with multiple applicable categories, tickets in unusual formats, tickets with very short or very long text, tickets containing technical jargon, and tickets from non-native English speakers. Your evaluation set should include examples from every category, with more examples from categories that are more frequent or where failure is more costly.

Edge case coverage. Edge cases are the inputs that are most likely to fail and most likely to be underrepresented in a dataset sampled from production traffic. They must be deliberately included. For a contract clause extractor, edge cases include contracts with missing standard clauses, contracts where the same clause appears in multiple places with inconsistent language, contracts in unusual formats, and contracts with non-standard terminology. For a question-answering system, edge cases include questions with ambiguous referents, questions that require multi-hop reasoning across documents, and questions where the correct answer is "the information is not available in the provided context." Include at least 15% to 20% edge cases in your evaluation set.

Labelling guidelines. Every example in your evaluation set needs a ground truth label — the reference answer or reference quality score that the eval will measure against. For classification tasks, the label is the correct category. For generation tasks, the label is either a reference output or a human quality rating, depending on your scoring approach. Labelling guidelines must specify: who is qualified to label each example type (domain knowledge requirements), how to handle ambiguous cases (a decision procedure, not "use your judgment"), what to do when multiple labels are plausible (pick one, label all, or flag as ambiguous), and the minimum agreement rate required across raters before a label is accepted.

Inter-annotator agreement. When multiple people label the same examples, their agreement rate measures how well-defined the labelling task is. For classification tasks, Cohen's kappa above 0.7 indicates acceptable agreement; below 0.6 suggests the categories or decision criteria need refinement. For generation quality tasks, agreement is inherently lower — human ratings on quality scales typically achieve kappa in the 0.4 to 0.6 range, which is acceptable but means you need more raters per example. Calculate inter-annotator agreement on a subset of your dataset before labelling the full set. If agreement is low, fix the guidelines before labelling more examples.

Why BLEU and ROUGE Fail for Open-Ended Generation

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are n-gram overlap metrics originally developed for machine translation and summarisation research. They compare a generated output to one or more reference outputs by measuring the overlap of word sequences (n-grams).

The n-gram overlap problem. "The policy was updated in 2024" and "The 2024 policy update was announced" contain roughly the same information but share very few n-grams. A paraphrase that preserves all the meaning but uses different vocabulary will score very poorly on BLEU/ROUGE, even though a human rater would judge it as equally correct. Conversely, "The policy was updated in 2024 and applies to all users" and "The policy was updated in 2024 and applies to no users" differ by one word but have nearly identical BLEU/ROUGE scores, even though one is factually wrong.

The reference dependency problem. BLEU and ROUGE measure similarity to a reference output. For open-ended generation tasks, a single reference output is rarely the only acceptable output. An LLM summarising a legal document can produce ten different valid summaries that a human would rate equally. Measuring BLEU against one of them penalises the others. Unless you have multiple human-written reference outputs per example (expensive to produce), n-gram overlap metrics systematically underestimate quality for open-ended tasks.

Where these metrics do apply. BLEU and ROUGE remain useful for constrained generation tasks where there is a small set of acceptable outputs and paraphrase variety is genuinely undesirable — some translation tasks, template-filling tasks, and code generation with exact expected outputs. For anything involving open-ended generation, flexible phrasing, or multi-sentence outputs, do not use these metrics as your primary quality signal.

Semantic Similarity Metrics

Semantic similarity metrics capture meaning overlap rather than surface-form overlap, making them more appropriate for open-ended generation tasks.

BERTScore. BERTScore computes token-level embeddings using a pretrained BERT model and measures the cosine similarity between tokens in the generated output and tokens in the reference output, using maximum similarity matching. Unlike n-gram metrics, BERTScore captures paraphrase similarity: two sentences that mean the same thing but use different words will have high BERTScore even if they share no n-grams. BERTScore correlates substantially better with human judgments than BLEU on summarisation and translation tasks. The limitation is that it still requires a reference output and therefore inherits the reference dependency problem.

Embedding cosine similarity. A simpler approach is to embed both the generated output and the reference output using a sentence embedding model (such as OpenAI's text-embedding-3-small or an open-source alternative), compute the cosine similarity of the resulting vectors, and use that as a quality proxy. This is fast to compute and works without a reference output if you are computing similarity to the input query or to retrieved context. For RAG systems, computing the embedding similarity between the generated answer and the retrieved context is a useful faithfulness proxy.

Semantic metrics as proxies, not verdicts. All automated semantic metrics are proxies for human judgment, not substitutes. A high BERTScore does not guarantee the output is correct — it guarantees that it is similar in embedding space to the reference, which is correlated with but not identical to human quality judgment. Use semantic metrics for tracking relative quality changes over time (regression detection) rather than for making absolute quality claims.

LLM-as-Judge Design

Using an LLM to score other LLM outputs — the LLM-as-judge approach — has become the practical standard for evaluating open-ended generation at scale, where human rating of every output is too expensive. Designing a reliable LLM judge requires addressing several known failure modes.

Scoring rubric construction. The quality of your LLM judge depends almost entirely on the quality of the scoring rubric you provide. A rubric that says "rate this response on accuracy from 1 to 5" will produce inconsistent, biased scores because "accuracy" is underspecified. A rubric that defines what each score means ("5: all claims in the response are directly supported by the provided context with no extrapolation; 4: all claims are supported but minor paraphrase or inference is present; 3: most claims are supported but one claim goes beyond the provided context; 2: multiple claims are not supported by the context; 1: the response primarily consists of claims not present in the context") produces calibrated, consistent scores that can be reproduced reliably.

Include three to five rubric anchor examples in your judge prompt: one example at each extreme of the scale, and examples at the ambiguous middle points. Anchor examples dramatically reduce score variability by giving the judge concrete calibration points rather than abstract descriptions.

Positional bias mitigation. LLM judges exhibit a positional bias: when asked to compare two responses (A vs. B), they tend to favour whichever response is presented first, regardless of quality. In pairwise comparison setups, run each comparison twice with the order flipped and flag cases where the judge changes its preference. If the judge's preference is unstable across orderings, treat the comparison as ambiguous rather than taking the first-position result.

For absolute scoring (rate this response on a 1–5 scale), positional bias is less relevant, but the judge can exhibit verbosity bias (preferring longer responses) and sycophancy bias (preferring responses that sound confident). Counteract verbosity bias by explicitly including in the rubric that length does not indicate quality. Counteract sycophancy bias by anchoring the rubric on verifiable criteria (supported by context, factually accurate) rather than on subjective impressions (well-written, compelling).

Self-evaluation bias. An LLM judge from the same model family as the model being evaluated tends to rate outputs from that family more favourably than outputs from other families. This is the self-evaluation bias. When you are running a cross-model comparison, use a judge from a different model family. When using LLM-as-judge for tracking quality within the same model family over time, this bias is less concerning because it is systematic — you are comparing the same model to itself, so the bias cancels.

Calibration against human labels. Before deploying an LLM-as-judge at scale, calibrate it: run the judge on a subset of examples that have been independently human-rated, and measure the correlation between judge scores and human scores. A Pearson correlation above 0.7 between judge and human ratings on your specific task indicates the judge is calibrated well enough to use as a quality proxy. A correlation below 0.6 means the judge rubric needs revision or the judge model needs to be reconsidered. Recalibrate periodically, especially after changing the judge model or rubric.

Metric Selection by Task Type

Different task types require different evaluation approaches. Using the wrong metric for a task type produces misleading quality signals.

Classification tasks. Use precision, recall, and F1 per class, not overall accuracy. Overall accuracy is dominated by the majority class and can obscure poor performance on minority classes. For multiclass classification, report metrics per class and identify which classes have lowest performance. For binary classification with different costs for false positives and false negatives (such as fraud detection or content moderation), define separate thresholds for precision and recall on the positive class.

Generation tasks. For constrained generation (fill-in templates, code generation with exact expected outputs), BERTScore or embedding cosine similarity against a reference output. For open-ended generation (summaries, explanations, recommendations), LLM-as-judge with a calibrated rubric is the practical choice at scale. Maintain a human-reviewed sample for calibration. Report both mean score and the distribution — specifically the tail (what percentage of outputs score below your quality threshold).

Retrieval tasks. For retrieval components in RAG systems, use precision at k (what fraction of the top k retrieved documents are relevant?), recall at k (what fraction of all relevant documents are in the top k?), and NDCG (normalized discounted cumulative gain, which rewards ranking relevant documents higher). These metrics require relevance judgments for each query-document pair, which must be human-labelled for your specific query set.

Reasoning tasks. For tasks that require multi-step reasoning, evaluate both the final answer and the reasoning chain. An LLM can arrive at a correct answer via incorrect reasoning, or produce a plausible-sounding but incorrect reasoning chain that arrives at a correct answer by coincidence. Process-level evaluation (is the reasoning valid step by step?) is more informative than outcome-only evaluation for reasoning tasks.