LLM Integration Patterns for Production Systems
Deliberate Academy Editorial Team
Reviewed for accuracy and professional relevance
You're 5 lessons in — don't lose your progress.
Sign up free to save where you are and earn a verified certificate when you pass.
- Identify the four primary LLM integration patterns and select the appropriate pattern for a described production use case
- Write production-quality system prompts with output format control and explain the prompt injection attack surface in LLM-integrated systems
- Implement output parsing and validation for LLM responses, including schema validation and retry logic for malformed output
- Design streaming response handling for web applications and explain when streaming improves user experience
- Specify error handling and fallback logic for the common LLM API failure modes: rate limits, timeouts, and model unavailability
Calling an LLM API from a prototype is straightforward. Building an LLM integration that is reliable, secure, cost-controlled, and debuggable in production is a different engineering challenge. Most of the gap between the two is in the patterns around the API call, not in the call itself.
This lesson covers the integration patterns that separate production LLM systems from demos.
The Four Primary LLM Integration Patterns
Pattern 1: Direct API call. The application sends a prompt to the LLM API and uses the response. This is appropriate when the task can be fully specified in a single prompt, the required context fits within the context window, and no external data retrieval is needed. Most simple generation, classification, and transformation tasks fit this pattern.
Pattern 2: Retrieval-augmented generation (RAG). The application retrieves relevant documents from an external store (a vector database, a keyword search index, or a hybrid of both), injects them into the context, and sends the enriched prompt to the LLM. This is appropriate when the task requires information not in the model training data, when the knowledge base changes frequently, or when you need the model to cite or reason about specific documents.
Pattern 3: Agent and tool use. The LLM is given access to tools (functions it can call) and iterates over multiple turns, calling tools and using their outputs to make progress on a task. This is appropriate for tasks that require multi-step reasoning, external data lookup at runtime, or actions that must be sequenced based on intermediate results. Tool use integrations are significantly more complex to build reliably and debug.
Pattern 4: Fine-tuning. A base model is further trained on domain-specific examples to adjust its behavior for a specific task. This is appropriate when prompting and RAG cannot achieve the required output style or format, when the task requires knowledge that cannot be efficiently injected via context, or when latency and cost requirements make shorter prompts (achievable via fine-tuning) necessary at scale. Fine-tuning is rarely the first option; it is the option you reach for after prompting and RAG have been exhausted.
Prompt Engineering in Code
System prompts are the foundation of every LLM integration. They specify the model's role, constraints, output format, and behavior. In production, system prompts are code: they should be version-controlled, reviewed, tested against your eval set, and changed intentionally.
System prompt structure. A production system prompt typically includes: role definition (what the model is and is not), task specification (precisely what it should produce), output format instructions (JSON schema, markdown structure, character limits), constraints (what topics or actions to avoid), and examples where the task is complex enough to benefit from few-shot demonstration.
Output format control. For structured output tasks, specify the exact format in the system prompt and use the provider's structured output or JSON mode feature where available. OpenAI's response format parameter with JSON schema enforcement significantly reduces malformed output rates. Anthropic's prompt patterns for structured XML output provide similar reliability. Specifying output format in the system prompt is necessary but not sufficient for production reliability.
Few-shot examples. For tasks where the desired output format or style is difficult to specify purely in instructions, two to five examples in the prompt dramatically improve consistency. Examples should cover representative cases including edge cases, not just the ideal case.
Prompt injection is a real attack surface in any LLM-integrated system that processes user-supplied text. If your application takes user input and includes it in an LLM prompt, a malicious user can include instructions in their input that override or modify your system prompt. For example, a customer service chatbot that summarises user complaints could be manipulated via the complaint text to exfiltrate system prompt contents, ignore constraints, or produce harmful outputs. Defences include: treating all user input as untrusted data and not interpolating it directly into instruction sections of the prompt, using structural separators between system instructions and user content, and validating outputs against expected schemas before acting on them.
Output Parsing and Validation
LLM output is not guaranteed to match the format you specified, even with JSON mode enabled. Production systems must validate output before using it.
Why validation is non-negotiable. LLMs can produce syntactically valid JSON that fails schema validation. They can produce output that is correctly formatted for 999 out of 1000 calls and malformed on the 1000th. They can produce output that passes schema validation but contains values that violate business logic constraints. Treating LLM output as trusted, validated data is a reliability risk.
Schema validation patterns. Define the expected output schema using Zod (TypeScript), Pydantic (Python), or a JSON Schema validator. Validate every LLM response against the schema before acting on it. If validation fails, log the failure with the full raw output and prompt context.
Retry logic for malformed output. A malformed output on the first call does not always mean the prompt is wrong. LLM output variance means a second call often produces a valid response. Implement retry logic with a maximum of two to three retries for validation failures, with the raw failed output logged for later analysis. If retries are exhausted and validation still fails, fall back to the defined fallback flow rather than surfacing raw LLM output to the user.
Structured output with provider features. OpenAI's structured outputs feature (response format with a strict JSON schema) enforces schema compliance at the API level, reducing (but not eliminating) the need for application-level validation. Anthropic's API supports guided generation patterns for structured output. Use these features when available; do not rely on them as a substitute for application-level validation.
Streaming Responses
LLM generation is sequential: tokens are produced one at a time. Streaming lets the client receive and display tokens as they are generated, rather than waiting for the complete response.
When to stream. Stream when the response is long enough that waiting for completion would be noticeable (generally more than 2 to 3 seconds of generation time) and when the use case is user-facing rather than background processing. Streaming is the standard expectation for chat interfaces. It is unnecessary and adds complexity for batch processing, classification, or any task where the output must be complete and validated before it is used.
Streaming in a web context. The standard approach is server-sent events (SSE) or WebSocket streaming from the server to the client. The server opens a streaming connection to the LLM API and forwards tokens to the client as they arrive. In Next.js, the Vercel AI SDK provides streaming utilities that handle the SSE boilerplate. The client renders tokens progressively as they arrive using React state or a streaming-aware library.
Streaming UX patterns. Display a loading indicator while waiting for the first token. Render tokens progressively as they arrive. Provide a stop generation control for long responses. For structured outputs that require complete parsing before rendering (JSON, code), buffer the stream and display a progress indicator rather than rendering partial structured data.
Error Handling and Fallbacks
LLM APIs fail in the same ways as any other external dependency, plus some additional AI-specific failure modes.
Rate limits. All providers impose requests per minute and tokens per minute limits. Rate limit errors (HTTP 429) require exponential backoff with jitter, not immediate retry. For batch workloads, implement a token-bucket rate limiter client-side to avoid hitting provider limits in the first place.
Timeout handling. LLM generation takes seconds. Your HTTP client timeout must be set accordingly: a 30-second timeout is too short for long-context requests on busy APIs; a 120-second timeout may be appropriate. Set connection and read timeouts separately and log timeout events with the prompt token count, since long prompts correlate with long generation times.
Model unavailability. Provider outages happen. Your AI feature should degrade gracefully when the API is unavailable: show the non-AI version of the feature, queue the request for later processing, or display a clear message that the AI feature is temporarily unavailable. Circuit breakers prevent cascading failures when the upstream API is degraded.
Cost control guardrails. Implement maximum token limits on both prompt and completion for every integration. An unbounded completion length on a high-traffic endpoint can produce runaway costs. Log token usage per call and set alerts for cost anomalies before they become expensive surprises.
Context Management in Long-Lived Applications
Applications with conversation history, long-running sessions, or iterative agent loops face a specific context management challenge: context grows without bound, eventually exceeding the context window and requiring truncation, which silently drops information.
Conversation history truncation. The naive approach (truncate from the beginning) drops the earliest context, which is often the most important (initial instructions, stated user goals, established constraints). Better approaches: summarise older turns into a compressed representation before truncating, use a sliding window that always preserves the system prompt and the most recent N turns, or store conversation history in a database and retrieve only the most relevant turns via semantic search.
Session state management. For long-running sessions, store conversation state in a database rather than in memory. This enables recovery from server restarts and provides an audit log of the full conversation for debugging purposes.
Agent loop context. Agent systems that call tools and iterate over multiple LLM turns can accumulate large context quickly: tool call requests, tool outputs, intermediate reasoning, and accumulated history. Set explicit context budgets for each component and implement truncation policies before context limits are reached, not after.
Context pollution from poorly chunked documents in a RAG system
Context
An engineering team built a RAG-based internal knowledge base assistant. Users could ask questions about company policies, processes, and technical documentation. The team chunked documents by splitting on paragraph breaks and embedded each chunk using OpenAI's text-embedding-3-small model. The assistant answered simple factual questions well in initial testing.
Action
After deployment, users began reporting inconsistent answers to questions that crossed section boundaries in the source documents. A question about the exception handling process for expense reimbursement returned a response that was accurate about the general process but missed the exception clause that was documented in a separate paragraph. Debugging revealed that the embedding similarity search was retrieving the correct high-level paragraph but not the exception clause paragraph, because the paragraphs had no semantic overlap in their embeddings. The team also found that some chunks contained headers and footers from PDFs that polluted the context with navigation artifacts, confusing the model's reasoning.
Outcome
The team implemented a hybrid chunking strategy: structural chunking that respected document section boundaries rather than paragraph breaks, a chunk overlap of 100 tokens to prevent context gaps at chunk edges, a preprocessing step to strip PDF navigation artifacts before embedding, and a reranking step using a cross-encoder model to reorder retrieved chunks by relevance before injection. After these changes, answer quality on multi-section questions improved substantially and user-reported inconsistencies dropped significantly.
A production LLM integration processes user feedback form submissions and classifies them into product issue categories. The system prompt instructs the model to respond only with a JSON object matching a specific schema. On 0.3% of calls, the model returns a valid JSON object that does not match the schema. What is the correct production approach?
Select one answer.
In the knowledge-base case study the assistant described the reimbursement process correctly but left out the exception clause. Why did retrieval miss that clause?
Select one answer.
Exercise
Your Task
Design the integration architecture for a described feature: a B2B SaaS product wants to add an AI feature that generates a weekly account health summary for each customer, drawing on the last 30 days of usage metrics stored in the application database. The summary should be 150 to 200 words, structured with three sections: usage highlights, areas for attention, and a recommended next action. The feature runs as a background job every Sunday night and emails the summary to the account manager on Monday morning. Specify: which integration pattern applies, the system prompt structure including output format, the validation approach, error handling for API failures during the batch job, and one potential failure mode that is specific to this use case.
Your reflection
Did you complete this exercise? What did you find? (Saved locally in your browser)
Try It: Run the Code
The exercise above is a system-design exercise. This exercise is different: it is deterministic, not AI-graded. Your code is actually executed in your browser and checked against fixed test cases — a direct, hands-on version of the output parsing and validation this lesson covers.
Parse and Validate an LLM's JSON Response
Per this lesson, LLM output is not guaranteed to match the schema you asked for, even with JSON mode enabled -- production systems must validate every response before using it. Complete `parse_llm_response(raw_text, required_fields)`. `raw_text` is the raw string returned by the model. `required_fields` is a list of field names the parsed object must contain. Your function must: 1. Try to parse `raw_text` as JSON. If parsing fails, return `{"valid": False, "error": "invalid_json"}`. 2. If the parsed value is not a JSON object (e.g. it is a list or a plain number), return `{"valid": False, "error": "not_an_object"}`. 3. Check that every name in `required_fields` is present as a key in the parsed object. If any are missing, return `{"valid": False, "error": "missing_fields", "missing": <sorted list of the missing field names>}`. 4. Otherwise, return `{"valid": True, "data": <the parsed object>}`.
- The four primary LLM integration patterns are direct API call, RAG, agent and tool use, and fine-tuning. Each has a defined appropriate use case. Fine-tuning is reached only after prompting and RAG have been exhausted.
- System prompts are code: version-controlled, reviewed, tested against eval sets, and changed intentionally. Output format instructions and few-shot examples are standard production system prompt components.
- Prompt injection is a real attack surface when user-supplied text is interpolated into prompts. Treat user input as untrusted data and separate it structurally from instruction sections.
- Validate every LLM response against the expected schema before acting on it. Implement retry logic for validation failures. Do not surface raw unvalidated LLM output to users or downstream systems.
- Rate limits, timeouts, and model unavailability are production failure modes that require explicit handling: exponential backoff, appropriate timeout configuration, circuit breakers, and graceful degradation to non-AI fallbacks.