Evaluating RAG Systems and Factual Accuracy

Retrieval-augmented generation systems fail in more ways than standard generation systems. A RAG system can fail because the retriever does not return the right documents. It can fail because the retrieved documents are correct but the generator ignores them. It can fail because the generator uses the retrieved documents but adds information from its parametric knowledge that contradicts them. It can fail in all three ways simultaneously. Diagnosing which failure mode is present requires component-level evaluation, not just end-to-end output quality measurement.

This lesson covers the evaluation frameworks specific to RAG systems and factuality verification, including the RAGAS framework, hallucination detection approaches, and the diagnostic patterns that identify where in a RAG pipeline errors originate.

The RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework that assesses RAG systems across four metrics, each measuring a distinct quality dimension of the pipeline. Understanding what each metric measures, and what a low score on each metric means diagnostically, is the starting point for RAG system evaluation.

Faithfulness. Faithfulness measures whether every claim in the generated answer is supported by the retrieved context. A faithful answer draws only on information present in the retrieved documents — it does not introduce facts from the model's parametric knowledge. Faithfulness is scored at the claim level: the answer is decomposed into atomic claims, and each claim is checked against the retrieved context to determine whether it is supported.

A low faithfulness score indicates that the generator is hallucinating — adding information beyond what the retrieved context provides. This is a generator failure mode, not a retriever failure mode. The retrieval step may have returned perfectly relevant documents; the generator simply chose to go beyond them.

Answer relevance. Answer relevance measures whether the generated answer actually addresses the user's question. A high-faithfulness answer can have low answer relevance if it accurately reports information from the retrieved context but does not address what was asked. Answer relevance is typically measured by generating hypothetical questions from the answer and measuring whether those generated questions are semantically similar to the original question.

A low answer relevance score indicates either a generator failure (the model is responding to a different question than the one asked) or a retriever failure (the retrieved context is about adjacent but not directly relevant topics, causing the generator to produce an answer that does not match the query).

Context precision. Context precision measures what fraction of the retrieved context is actually relevant to the question. If your retriever returns 10 chunks and 3 of them are relevant while 7 are noise, context precision is 0.3. High context precision means the retriever is returning focused, relevant results. Low context precision means the retriever is returning too many irrelevant chunks, which dilutes the context and can confuse the generator.

Low context precision is a retriever failure. The fix is retrieval quality improvements: better chunk embedding, reranking, or tighter retrieval parameters.

Context recall. Context recall measures what fraction of the information needed to answer the question is present in the retrieved context. A retriever can have high precision (everything it returns is relevant) but low recall (it missed many relevant documents). Context recall requires ground truth labels specifying which information is required to answer each question — it cannot be computed without annotation.

Low context recall is a retriever failure of a different type than low precision. Low precision means the retriever is returning irrelevant documents. Low recall means the retriever is missing relevant documents entirely.

Building a RAGAS Evaluation Harness

A RAGAS evaluation harness applies these four metrics across your evaluation dataset. The practical implementation has four components: query set, reference answers, RAG pipeline outputs, and scoring.

The query set. Your evaluation queries should cover the full range of question types your production system will encounter: factual lookups (what is X?), procedural questions (how do I do X?), comparative questions (what is the difference between X and Y?), and questions that require synthesising information from multiple documents. Include queries where the correct answer is not in the knowledge base — these test whether the system correctly handles out-of-scope queries rather than hallucinating an answer.

Reference answers. RAGAS metrics that require ground truth (context recall and answer relevance) need reference answers: the correct answer to each query, and the set of documents or document chunks that are required to answer it. These reference answers must be human-generated; using the RAG system itself to generate reference answers creates circular evaluation.

Running the harness. For each query in the evaluation set, run the full RAG pipeline and capture both the retrieved context and the generated answer. Apply RAGAS scoring to each query-context-answer triple. Aggregate scores across the full evaluation set, and also compute scores by query category — performance differences across categories often reveal systematic retrieval or generation weaknesses on specific question types.

Threshold setting. Define minimum acceptable scores for each RAGAS metric, calibrated to your use case. A customer-facing FAQ system that will be used by patients querying medical information might require faithfulness above 0.90 (very little tolerance for hallucination). An internal research assistant where users understand the outputs are model-generated might accept faithfulness of 0.80. Set thresholds based on the consequence of failures in your specific context, not on what the system currently achieves.

Hallucination Detection

Faithfulness measurement at the aggregate level tells you how often the system hallucinates. Hallucination detection at the instance level tells you which specific claims in a specific output are not grounded in the context. Both are needed: aggregate metrics for quality monitoring, instance-level detection for flagging outputs before they reach users.

NLI-based claim verification. Natural Language Inference (NLI) models are trained to classify whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise. In hallucination detection, the retrieved context is the premise and each claim extracted from the generated answer is the hypothesis. Claims classified as entailed by the context are grounded; claims classified as contradicted or neutral are potential hallucinations.

NLI-based verification is fast and cheap to run but has limitations: NLI models trained on general text may be miscalibrated on domain-specific content, and the claim extraction step (splitting the answer into atomic claims) is itself error-prone. Use cross-encoder NLI models trained on evidence-based tasks for better calibration than general NLI models. Models such as the HHEM (Hughes Hallucination Evaluation Model) family are specifically designed for hallucination detection.

LLM-based claim verification. An alternative approach is to use an LLM as the verifier: extract each claim from the generated answer, then ask the LLM whether each claim is supported by the retrieved context. This approach is more flexible than NLI models and handles complex claims better, but it is slower and more expensive. The verification prompt must be carefully designed to prevent the LLM from using its parametric knowledge to evaluate claims — the LLM should be instructed to answer solely based on the provided context, and examples in the prompt should reinforce this constraint.

Combining approaches. For high-stakes applications, use a pipeline that first applies NLI-based claim verification (fast, cheap, catches clear hallucinations) and then applies LLM-based verification to claims the NLI model rated as neutral rather than entailed — the ambiguous middle ground where the NLI model lacks confidence. This hybrid approach balances cost and coverage.

Diagnosing Where Errors Originate: Retriever vs. Generator

The most important diagnostic question in RAG evaluation is whether quality failures are retriever failures or generator failures. The answer determines where to invest improvement effort.

The component isolation approach. To diagnose the error source, evaluate the retriever and generator separately, not just end-to-end.

For retriever evaluation: given the same query, how often does the retriever return at least one document that contains the information needed to answer the question? Compute retrieval recall (does the correct information appear anywhere in the retrieved context?) and precision (how much of the retrieved context is relevant?). If retrieval recall is high but end-to-end answer quality is low, the failure is in the generator. If retrieval recall is low, the failure is in the retriever regardless of generator performance.

For generator evaluation: given the same query and the retrieved context that was actually returned (not ideal context), how faithfully and relevantly does the generator use it? Compute faithfulness and answer relevance on the actual pipeline outputs. If faithfulness is low, the generator is going beyond the context. If answer relevance is low, the generator is not focusing on the query.

The oracle retrieval test. A powerful diagnostic is to run the generator with oracle context — the ideal, human-curated context that perfectly answers the question — rather than the context returned by the retriever. If end-to-end quality with oracle context is high, the problem is retrieval. If end-to-end quality remains low even with oracle context, the problem is the generator or the prompt. This test cleanly separates the two failure modes.

The empty context test. A useful complement is to run the generator with no retrieved context and measure how often it produces answers that would be indistinguishable from its full-pipeline outputs. If the generator produces similar-quality answers without retrieval, it may be ignoring the retrieved context entirely and relying on parametric knowledge — which creates a faithfulness risk even when retrieval is working correctly.

End-to-End RAG Evaluation Pipeline

A production RAG evaluation pipeline integrates component-level and end-to-end evaluation into a repeatable harness that runs on a schedule and on every system change.

The pipeline has five steps. First, run the full RAG system on the evaluation query set and capture all intermediate outputs: query, retrieved chunks (with retrieval scores), generated answer, and any reranking or filtering steps. Second, run component-level retrieval evaluation: compute precision at k and recall at k for each query using human-labelled relevance judgments. Third, run end-to-end generation evaluation: compute faithfulness, answer relevance, and a custom accuracy metric (if you have reference answers) on each query-context-answer triple. Fourth, aggregate all metrics and compare to baseline and quality thresholds. Fifth, log detailed results per query to a persistent store so that per-query quality can be tracked over time and regressions can be diagnosed at the example level.

Tools that provide eval harness infrastructure specifically designed for RAG systems include RAGAS (open source Python library), Braintrust's RAG evaluation suite, and LangSmith's dataset and eval management platform. All three support automatic computation of the core RAGAS metrics and integration with your existing RAG pipeline.