CI/CD Pipelines for AI-Powered Features

Shipping code through a CI/CD pipeline that runs tests and blocks deployment on failures is standard engineering practice. Shipping AI features through the same pipeline — with eval runs that block deployment when quality regressions are detected — is the standard that separates teams that catch AI regressions before production from those that catch them after.

The challenge is practical: LLM eval runs are slower and more expensive than unit tests, they produce probabilistic results rather than binary pass/fail, and they require thoughtful cost management to run sustainably in CI. This lesson covers how to design and implement CI quality gates for AI features that are rigorous enough to catch real regressions but practical enough to run on every pull request.

Eval-on-PR Quality Gates: Blocking vs. Non-Blocking

A quality gate in CI blocks a pull request from merging until the specified condition is met. For AI features, the question of whether an eval gate should be blocking or non-blocking depends on the nature of the change and the cost of a regression.

Blocking gates prevent merging when a quality regression is detected. They are appropriate for: prompt file changes, model version changes, eval set changes, and any change to the code that constructs the prompt or processes the model output. A blocking gate on prompt changes is the minimum safety bar — it ensures no prompt modification reaches production without passing a quality check.

The blocking threshold should be set at the quality floor, not the current baseline. If your feature currently achieves a mean accuracy score of 4.3 but the committed quality floor is 4.0, the blocking gate should trigger when the eval run produces a score below 4.0, not when it produces a score below 4.3. A drop from 4.3 to 4.1 is a quality change worth investigating but not a committed-quality violation.

Non-blocking gates run eval checks but report results without blocking the merge. They are appropriate for: refactoring changes that are not expected to affect prompt or model behaviour, infrastructure changes, and changes to monitoring or logging code. Non-blocking gates still provide value — the eval results are available for review and can be inspected manually — but they do not slow down infrastructure changes with AI eval overhead.

Designing the gate decision. Map each file path pattern to a gate type in your CI configuration. Prompt files, model configuration files, and output parsing code trigger blocking gates. All other paths trigger non-blocking gates or skip the eval entirely. This ensures the eval runs when it matters and does not create unnecessary overhead for changes that cannot affect AI output quality.

Configuring Eval-on-PR in GitHub Actions

A basic eval-on-PR workflow in GitHub Actions runs on pull request creation and update, detects whether any AI-relevant files have changed, runs the eval suite if they have, and reports results as a PR check.

name: AI Eval Gate

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/lib/ai/**'
      - 'src/data/evals/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install eval dependencies
        run: pip install -r eval/requirements.txt

      - name: Run eval suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
        run: python eval/run_eval.py --suite regression --output eval-results.json

      - name: Check quality threshold
        run: python eval/check_threshold.py --results eval-results.json --config eval/thresholds.yaml

      - name: Post results to PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('eval-results.json'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `Eval results: mean accuracy ${results.mean_accuracy}, failure rate ${results.failure_rate}%`
            });

The check_threshold.py script reads the eval results and exits with code 1 if any metric is below the threshold defined in eval/thresholds.yaml. GitHub Actions treats a non-zero exit code as a step failure, which marks the CI check as failed and prevents merging if the check is marked as required.

Results comparison across runs. A single eval result in isolation is less informative than a comparison against the previous run. CI integrations for LangSmith and Braintrust support this natively: you can tag each eval run with the PR branch and compare to the baseline run (main branch) to see the delta for each metric, not just the absolute score. This comparison is the most actionable output of an eval-on-PR gate — engineers can see "this prompt change improved mean accuracy by 0.2 but increased the failure rate on edge cases by 3 percentage points" rather than just seeing whether the change passed a binary threshold.

Managing Eval Costs in CI

Running a full LLM eval on every pull request can become expensive. A 200-example eval set running at $0.01 per call (LLM API cost) plus $0.005 per call (judge API cost) costs approximately $3 per run. At 50 pull requests per week, that is $150 per week — manageable. At 500 pull requests per week across multiple teams, it becomes $1,500 per week, which is significant.

Sampling strategies. Rather than running the full eval suite on every PR, run a sampled subset on every PR and the full suite on merges to main. A 50-example sampled suite costs 25% of the 200-example full suite. The 50-example subset should be stratified to include examples from all input categories, including edge cases — random sampling without stratification may undersample the categories where regressions are most likely.

Fast and slow eval tiers. Design your eval suite in two tiers. The fast tier (50 to 100 examples, automated scoring only) runs on every PR within 5 to 10 minutes and catches clear regressions cheaply. The slow tier (full set, 200 to 500 examples, including LLM-as-judge scoring) runs on merges to main or on a daily schedule and provides the full quality picture. Blocking gates are attached to the fast tier; the slow tier provides additional visibility and catches subtler regressions that the fast tier might miss.

Prompt caching in test environments. Most major LLM providers support prompt caching — if the system prompt prefix is identical across calls, the provider caches the prefix computation and charges reduced rates for subsequent calls. This makes eval runs with the same system prompt significantly cheaper than calls with variable system prompts. Structure your eval runner to use the same system prompt across all test calls (which it should be doing anyway, since you are testing one prompt version at a time) and enable prompt caching in your API calls. OpenAI's cached tokens are charged at approximately 25% of the standard input token rate; Anthropic's cache write tokens are charged at 125% with a 90% discount on subsequent cache reads.

Model version pinning in test. Pin the model version used in eval runs, even if you are testing a prompt change against a model version alias (such as gpt-4o-latest) in production. Pinning the eval model ensures that the eval run on Monday and the eval run on Tuesday measure against the same model, producing comparable results. An unpinned eval that compares a new prompt against a different model version than the previous eval run is measuring two things at once.

LangSmith CI Integration

LangSmith provides a CI integration that makes eval results first-class citizens in the pull request workflow. The integration supports running evals against datasets stored in LangSmith, comparing results to a baseline, and posting the comparison summary to the PR.

Dataset management in LangSmith. Store your evaluation dataset in LangSmith as a named dataset. Each dataset version is immutable — adding new examples creates a new version rather than modifying the existing one. This means your eval results are always comparable to the correct dataset version and you can track quality across dataset versions as the eval set evolves.

Running evals in CI via LangSmith. The LangSmith Python client supports creating eval runs programmatically:

from langsmith import Client
from langsmith.evaluation import evaluate

client = Client()

def run_pipeline(inputs):
    # your RAG or generation pipeline
    return {"output": generate_response(inputs["question"])}

results = evaluate(
    run_pipeline,
    data="my-eval-dataset",
    evaluators=[accuracy_evaluator, faithfulness_evaluator],
    experiment_prefix="ci-run",
    metadata={"pr_number": os.environ.get("PR_NUMBER")}
)

LangSmith records the run, evaluator scores, and metadata, and makes the results available via the API or the web UI. The CI step can then compare this run's scores to the baseline (the last passing main branch run) and fail if any metric shows a statistically significant regression.

Braintrust CI Integration

Braintrust is an eval platform built specifically for CI-first evaluation workflows. Its CLI tool (braintrust eval) is designed to run in GitHub Actions and supports experiment comparison natively.

Experiment management. Braintrust organises eval runs as experiments within a project. Each experiment is tagged with the git commit, branch, and custom metadata. Comparing two experiments (for example, the current PR versus the baseline on main) is a native operation in the Braintrust UI and API.

CI workflow with Braintrust. The Braintrust CLI integrates into GitHub Actions with a single command that runs your eval script, pushes results to Braintrust, and prints a summary that can be captured and posted to the PR. A failed experiment (where a metric is below threshold) causes the CLI to exit with a non-zero status code, which fails the CI step.

Eval-driven development workflow. The most disciplined use of CI eval integration is the eval-driven development workflow: write or update the evaluation set before changing the prompt, run the eval against the current prompt to establish a baseline, make the prompt change, run the eval against the new prompt, and only merge if the change meets or exceeds the baseline on all tracked metrics. This workflow treats eval results as the definition of done for prompt development, not as a post-hoc validation step.