Building Production LLM Systems Capstone Exercise — Building Production LLM Systems

What you'll learn

Design a complete RAG architecture — chunking strategy, embedding model selection, vector store selection, and retrieval strategy — for a multi-format, access-controlled enterprise knowledge base
Design an output quality and reliability pipeline covering structured citation output, an evaluation framework mapped to RAGAS metrics and an LLM-as-judge rubric, and defence-in-depth confidentiality enforcement
Build a cost and performance architecture for a production LLM system, including token cost modelling with prompt caching, model routing, and a per-stage latency budget under a concurrency constraint
Design a production observability system — trace instrumentation, automated quality monitoring, and model deprecation risk management — for a multi-step LLM pipeline

This capstone asks you to design a production LLM architecture from end to end. It is not a coding exercise — it is a systems design exercise that requires you to synthesise the architectural, engineering, and quality decisions covered across all nine lessons.

Read the scenario carefully, then complete the exercise before reviewing the self-assessment criteria.

Scenario: Enterprise Knowledge Assistant

Context. A 1,200-person professional services firm — management consultants and industry specialists — wants to build an internal knowledge assistant. Their knowledge base consists of:

8,000 project reports and case studies (PDF format, 10-40 pages each, proprietary client data inside)
15,000 internal methodology documents and playbooks (Markdown and Word, well-structured)
2,400 proposal templates and winning proposals (PDF format, variable structure)
A continuously updated stream of 300 new documents per week

The assistant will be used by consultants via a web interface during client work. Typical queries include: finding relevant precedents ("have we done a cost optimisation engagement in automotive before?"), retrieving specific methodology sections ("what is our standard approach to stakeholder mapping?"), and synthesising information across documents ("what themes appear across our energy sector case studies?").

Requirements from the business:

Responses must cite the specific documents and sections they draw from
Data must never leave the firm's own cloud infrastructure (strict data residency)
Response latency must be under 4 seconds for 90% of queries
Extraction of client names from project reports must be suppressed in all responses (confidentiality)
The system must handle 200 concurrent users at peak load
Quality must be measurable and must not degrade silently over time

Capstone Exercise

Design a Production RAG Architecture for a Confidentiality-Constrained Enterprise Knowledge Assistant

Context

A 1,200-person professional services firm wants an internal knowledge assistant over 8,000 client project reports and case studies (PDF, 10-40 pages, proprietary client data inside), 15,000 well-structured internal methodology documents and playbooks, and 2,400 variably structured proposal templates, with 300 new documents added per week. Consultants will query it via a web interface for precedent lookups, methodology retrieval, and cross-document synthesis. The business requires: responses that cite the specific documents and sections they draw from, strict data residency (nothing leaves the firm's own cloud infrastructure), sub-4-second response latency for 90% of queries, suppression of client names from all responses, support for 200 concurrent users at peak load, and quality that is measurable and does not degrade silently over time.

Your Task

Design the complete production LLM architecture in four parts. (1) RAG Architecture Design: specify your chunking strategy per document type given the client-sensitive content in project reports, your embedding model choice under the self-hosting and data-residency constraint, your vector database choice given the data residency and role-based access requirements, whether you use vector-only, hybrid BM25-plus-vector, or sparse-dense fusion retrieval with reranking, and how you assemble the final context, including client-name suppression and citation metadata. (2) Output Quality and Reliability Design: define a structured JSON citation schema (source document, section, page range) enforced at the provider level and validated at the application layer; design an evaluation framework specifying golden dataset composition, which RAGAS metrics map to citation accuracy and answer completeness, your LLM-as-judge rubric for quality dimensions RAGAS does not cover, and how evaluation plugs into deployment; and design a defence-in-depth approach to client-name suppression that does not rely solely on the model following instructions, combining pre-retrieval filtering, post-generation redaction, and an audit log. (3) Cost and Performance Architecture: calculate the per-query token cost assuming a 3,000-token cached system prompt, 5 retrieved chunks averaging 600 tokens each, a 150-token user query, and a 400-token response, with the applicable Anthropic or OpenAI caching discount applied; specify model routing across query classification, retrieval-based responses, and synthesis responses against the 4-second p90 latency requirement; break the latency budget into embedding, retrieval, reranking, generation, and post-processing stages and identify the binding constraint at 200 concurrent users; and design the ingestion pipeline for 300 new documents per week, including format-specific extraction and synchronous-versus-asynchronous processing. (4) Observability and Quality Assurance: specify the full trace event list per query and how events group into a session replay; design production quality monitoring including automated scoring coverage, alerting thresholds, and a specific citation-accuracy metric; and define a model deprecation risk management plan covering version pinning, drift detection on pinned versions, and the migration procedure when a deprecation is announced.

Your notes (optional)

Deliverable

A structured, four-part production architecture design (RAG architecture, output quality and reliability, cost and performance, observability and QA) specific to the firm's document mix and constraints, with concrete parameters — chunk sizes, model choices, token costs, latency budgets, alert thresholds — rather than generic recommendations. Review your design against the self-assessment criteria below before you consider it complete.

Self-Assessment Criteria

Review your designs against these criteria before submitting for the exam.

Retrieval architecture:

Chunking strategy is specific to each document type, not a single generic approach
Embedding model is self-hostable (data residency compliance)
Vector store selection accounts for both data residency and multi-tenant access control
Hybrid search is justified or explicitly rejected with reasoning
Client name suppression is addressed at the retrieval layer, not only in generation

Output quality:

Structured citation schema is defined with specific field types
Evaluation dataset composition includes consultant-specific query types
RAGAS metrics are matched to the specific quality concerns (citation accuracy maps to faithfulness, answer completeness maps to context recall)
Confidentiality enforcement uses multiple independent layers, not a single LLM instruction

Cost and performance:

Token cost estimate accounts for prompt caching correctly
Routing decision addresses the 4-second latency constraint specifically
Latency budget breaks down to individual pipeline stages, not just an overall number
Ingestion pipeline specifies async handling for the 300-document weekly volume

Observability:

Trace specification captures data at each pipeline stage, not just the LLM call
Quality monitoring includes automated scoring on a production sample, not just golden dataset evaluation
Citation accuracy is monitored as a specific metric, not only as part of general quality
Model deprecation risk management is addressed proactively, not reactively

Going Further

If you want to extend this exercise, consider:

Security extension: The system handles confidential client data. What additional security controls would you add to the API layer, the vector store, and the trace storage? How would you implement role-based access control so junior consultants cannot retrieve documents from engagements they were not part of?

Continuous learning extension: How would you use production query data to improve the system over time? What feedback mechanisms would you add to the UI? How would you use consultant feedback to expand the golden dataset?

Multi-modal extension: Project reports often contain charts, diagrams, and exhibits that contain relevant information not captured in text extraction. Design the extension to the architecture that would enable retrieval of visual content from project reports.

Quick check

What does this capstone require of a defence-in-depth design for client-name suppression?

Select one answer.

Key takeaways

A production RAG architecture is a chain of interacting decisions, not independent ones — chunking strategy determines what an embedding model can retrieve well, and both interact with your latency and cost budget under real concurrency.
Confidentiality and compliance requirements, such as data residency and client-name suppression, belong at the architecture layer, not as a prompt instruction bolted onto an otherwise generic RAG design. Defence-in-depth means at least one control that does not depend on the model behaving correctly.
Token cost and latency are architectural constraints, not afterthoughts: prompt caching strategy, model routing by query type, and per-stage latency budgeting determine whether a design that works in a demo survives 200 concurrent users in production.
Observability for a multi-step LLM pipeline must attribute a quality or latency problem to a specific stage — embedding, retrieval, reranking, or generation — not just flag that something degraded.
Model deprecation is a planned event, not a surprise: pinning versions without a monitoring and migration plan only delays the incident instead of preventing it.