LLM Evaluation Frameworks

The standard software development feedback loop — write code, run tests, check pass/fail — does not transfer cleanly to LLM systems. LLM outputs are not deterministic and cannot be evaluated with exact-match assertions. The challenge is not whether the output matches an expected string, but whether it is good: accurate, relevant, faithful to the source, appropriately concise, and aligned with the user's actual need.

Most teams skip formal evaluation until quality problems force them to. By then, they have no baseline to compare against, no systematic way to measure improvement, and no way to detect whether a change helped or hurt. Evaluation built before problems arise is categorically more valuable than evaluation built in response to them.

Why Traditional Metrics Fail

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) were developed for machine translation and summarisation research in the early 2000s. Both measure n-gram overlap between a generated output and a reference text. A high BLEU score means the generated text uses many of the same word sequences as the reference; a high ROUGE score means the reference's word sequences appear in the generated text.

These metrics fail for production LLM evaluation for three reasons:

Surface form dependency. Two sentences that are semantically identical but use different words score zero n-gram overlap. "The contract terminates on December 31st" and "The agreement ends at year-end" express the same fact but share almost no n-grams. BLEU would rate one as wrong even if both were correct answers.

Reference dependency. BLEU and ROUGE require a gold-standard reference text to compare against. For most production LLM tasks, there is no single correct answer — there are many acceptable phrasings of a correct answer, and the "correct" phrasing depends on context. Collecting single-reference gold standards for production queries is expensive and produces misleading metrics.

Quality dimension blindness. BLEU measures n-gram precision; ROUGE measures n-gram recall. Neither measures factual accuracy, logical coherence, faithfulness to source documents, appropriateness of tone, or any of the quality dimensions that actually matter for most production LLM applications.

LLM-as-Judge Design

LLM-as-judge uses a separate LLM call to score the quality of a generated output. This approach can evaluate quality dimensions that string-matching metrics cannot, is scalable to large evaluation sets, and can be designed to approximate human evaluator judgements.

Scoring rubric design. An LLM judge needs an explicit scoring rubric — a description of what each score means — to produce consistent, calibrated ratings. Vague rubrics produce inconsistent scores. Effective rubrics are multi-level with concrete distinguishing criteria:

• 1 (Poor): The response contains factual errors, contradicts the source material, or fails to address the question.
• 2 (Marginal): The response addresses the question but omits important context or contains minor inaccuracies.
• 3 (Acceptable): The response correctly addresses the question with sufficient detail and no factual errors.
• 4 (Good): The response correctly addresses the question, provides appropriate context, and is clearly phrased.
• 5 (Excellent): The response fully addresses the question, anticipates follow-up information the user needs, and is concisely phrased.

Prompt structure for judges. A well-structured judge prompt includes: the original user query, the source documents or context used to generate the response (for faithfulness evaluation), the response being evaluated, the scoring rubric, and an instruction to produce a score and a brief justification. The justification is as important as the score: it makes the judge's reasoning auditable and reveals calibration issues.

Known biases and mitigations. LLM judges exhibit systematic biases: length bias (preferring longer responses), position bias (preferring the first option in pairwise evaluation), self-enhancement bias (preferring responses that resemble the judge model's own generation style). Mitigations include: using a different model as the judge than the one generating responses, double-evaluating pairwise comparisons in both orderings and averaging the results, maintaining a human-rated calibration set to detect systematic divergence.

RAGAS for RAG Evaluation

RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework specifically designed for RAG pipelines. It measures four dimensions that together characterise RAG system quality without requiring human-annotated reference answers for most metrics.

Faithfulness measures whether all statements in the generated answer can be inferred from the retrieved context. An answer that makes claims not supported by the retrieved documents is unfaithful, regardless of whether those claims happen to be correct. Faithfulness is critical for RAG systems because retrieved context is the intended source of truth — unfaithful answers indicate the model is hallucinating beyond the retrieved evidence.

Implementation: RAGAS decomposes the generated answer into individual statements, then asks an LLM to evaluate whether each statement is entailed by the retrieved context. Faithfulness = (number of supported statements) / (total statements). A faithfulness score below 0.8 typically indicates systematic hallucination beyond retrieved context.

Answer relevance measures whether the generated answer actually addresses the user's question. A response that is fully faithful to the retrieved context but answers a different question than the one asked has low answer relevance. Implementation: RAGAS generates synthetic questions from the generated answer and measures how similar those synthetic questions are to the original question.

Context precision measures whether the retrieved context is precisely targeted at the question — that is, whether the retrieved chunks are relevant to the question rather than tangentially related. Low context precision means the retrieval system is returning partially relevant or irrelevant chunks alongside the genuinely relevant ones. Implementation: for each retrieved chunk, an LLM rates whether it is useful for answering the question.

Context recall measures whether the retrieved context contains all the information needed to answer the question completely. This metric requires ground truth answers — it evaluates whether each statement in the reference answer can be attributed to the retrieved context. Low context recall means the retrieval system is missing key information that should have been retrieved.

Using RAGAS in practice. Context precision and faithfulness are the most actionable metrics because they do not require ground truth answers and can be run on any production query sample. Context recall and answer relevance provide additional diagnostic signal but require more setup. Run RAGAS on a sample of 100 to 200 production queries weekly to track trends, and on a fixed golden question set to detect regressions.

Reference-Based vs. Referenceless Evaluation

Reference-based evaluation compares the generated output against a human-provided reference answer. It is accurate for tasks with well-defined correct answers (fact extraction, structured output generation, code generation) and is the most defensible evaluation methodology. The limitation is cost: collecting high-quality reference answers for every query type is expensive and requires domain expertise.

Referenceless evaluation (which RAGAS mostly implements) evaluates quality from the generation context alone — without requiring a pre-annotated reference. It is scalable to large query sets and can run on production traffic. The limitation is that referenceless metrics can be fooled by coherent but wrong responses: an answer that is faithfully grounded in incorrect retrieved documents scores well on faithfulness but is factually wrong.

The practical recommendation is to use both: a small (50 to 200 query) human-annotated golden dataset with reference answers for regression testing, and referenceless LLM-as-judge and RAGAS metrics on larger samples of production traffic for monitoring. Neither alone is sufficient; together they provide complementary coverage.

Golden Dataset Curation

A golden dataset is a fixed set of inputs with human-verified expected outputs (or quality ratings) that you evaluate against repeatedly to track performance over time. Curating a golden dataset well is the single most valuable investment in LLM system quality.

Selection. The dataset should include: representative examples from each query category your system handles, edge cases that have caused failures in production or testing, adversarial inputs that probe known failure modes, and recent additions that reflect the current query distribution. Do not sample purely randomly — a random sample of production queries will under-represent rare but important query types.

Labelling. Labels must come from qualified annotators — domain experts for specialised tasks, not crowdworkers. For structured output tasks, the label is the correct structured output. For quality evaluation tasks, the label is a human quality rating on a defined rubric (the same rubric you will use for LLM-as-judge calibration). Collect multiple annotations per item for high-stakes evaluations and compute inter-annotator agreement.

Maintenance. Golden datasets become stale as the system evolves. Review quarterly: remove examples that no longer represent current usage, add examples that cover new query types or failure modes discovered since the last review. Track dataset version alongside evaluation results so you can distinguish performance changes caused by dataset changes from performance changes caused by system changes.

Eval-Driven Development in CI/CD

Eval-driven development means that every change to a prompt, model, retrieval configuration, or generation pipeline is evaluated against the golden dataset before merging, and merges are blocked if the evaluation score drops below a defined threshold.

CI/CD integration. A typical integration: the CI pipeline triggers on pull requests that touch any file in the LLM integration (prompts, retrieval code, generation code). The eval job runs the golden dataset through the changed pipeline, computes the standard metrics, and compares against the last passing baseline. If any metric drops more than 2-3% below baseline, the pipeline fails and the PR is blocked.

Evaluation job cost control. Running a full evaluation suite on every PR can be expensive if the golden dataset is large or uses expensive models. Mitigations: run a smaller fast evaluation tier on every PR and the full evaluation suite only on merges to main; use cheaper models for LLM-as-judge in CI (accepting some calibration loss) and expensive models for production quality gates; parallelize eval jobs to minimise wall-clock time.

Baseline tracking. Store evaluation results indexed by commit hash and dataset version. This enables a rolling baseline view: not just "did this PR regress?" but "over the last 30 commits, is quality trending up or down?" Quality drift that is imperceptible per-PR can be significant over a longer window.