Structured Output and Production Validation

Every production LLM integration that produces structured data faces the same core challenge: the model generates text, and text is inherently unstructured. Even with the most precisely written system prompt, JSON mode enabled, and a strict schema, LLM output is not a typed API response. It is a string that you hope conforms to the expected structure.

This distinction matters because downstream code that consumes structured LLM output is almost always written assuming the structure will be correct. When the structure is not correct — and it will not be on some fraction of calls — unvalidated downstream code breaks in ways that are difficult to debug and may produce incorrect behaviour silently rather than failing loudly.

The solution is a validation pipeline that sits between the raw LLM response and any downstream code that consumes it. This lesson covers how to build that pipeline correctly.

JSON Schema Enforcement at the Provider Level

Both major LLM providers offer mechanisms to enforce structured output at the API level, reducing (but not eliminating) the rate of schema violations.

OpenAI structured outputs (released mid-2024, now standard) allow you to pass a JSON Schema object as the response_format parameter with strict: true. When strict mode is enabled, the API uses constrained decoding to enforce schema compliance at token generation time. The model's token sampling is restricted so that each token is guaranteed to produce a valid continuation of the schema. This virtually eliminates JSON parsing errors and schema structure violations for schemas that are fully supported.

Important limitations: strict mode does not validate semantic constraints (a field declared as a string can still receive any string value), does not validate against business logic rules (a price field that must be positive can still receive a negative number), and requires that the schema be a strict subset of JSON Schema — recursive schemas and some advanced JSON Schema features are not supported. Strict mode also adds a small inference overhead.

Anthropic tool use for structured output works differently: instead of using a response format parameter, you define the desired output structure as a tool with a JSON Schema input definition, and instruct the model to call that tool with the structured data. The model is constrained to produce output that matches the tool's input schema. This approach is more reliable than asking the model to produce JSON in a plain message.

The practical difference: for OpenAI, use response_format with JSON Schema and strict: true. For Anthropic, define a tool called something like submit_result with your schema as its input, and instruct the model to call it. Both approaches reduce schema violations significantly but do not eliminate them entirely.

The Instructor Library Pattern

Instructor is a Python library that wraps the OpenAI, Anthropic, and other provider clients to make structured output with Pydantic validation the default workflow rather than an afterthought.

The core pattern: define your expected output as a Pydantic model. Pass the model as the response_model parameter to the Instructor-patched client. Instructor handles the schema generation, the provider-specific structured output configuration, the response parsing, and the retry logic.

import instructor
from anthropic import Anthropic
from pydantic import BaseModel, Field

class ExtractionResult(BaseModel):
    company_name: str
    revenue_usd: float = Field(gt=0, description="Annual revenue in USD, must be positive")
    fiscal_year: int = Field(ge=2000, le=2030)
    confidence: float = Field(ge=0.0, le=1.0)

client = instructor.from_anthropic(Anthropic())

result = client.messages.create(
    model="claude-3-7-sonnet-20250219",
    max_tokens=1024,
    response_model=ExtractionResult,
    messages=[{"role": "user", "content": document_text}]
)

Instructor handles: generating the JSON Schema from the Pydantic model, sending it to the provider as a tool definition or structured output parameter, parsing the response, running Pydantic validation (including Field constraints), and retrying with validation error feedback if validation fails.

Pydantic validators for business logic. Pydantic's @field_validator and @model_validator decorators allow you to add arbitrary Python validation logic that runs after type checking. For LLM output validation, this is where you enforce business logic constraints that cannot be expressed in JSON Schema:

from pydantic import BaseModel, field_validator, model_validator
from datetime import date

class DateRange(BaseModel):
    start_date: date
    end_date: date

    @model_validator(mode='after')
    def end_after_start(self) -> 'DateRange':
        if self.end_date <= self.start_date:
            raise ValueError('end_date must be after start_date')
        return self

Instructor will include the validation error in the retry message to the model, which often allows the model to correct its output on the next attempt.

TypeScript and Zod Patterns

For TypeScript backends, Zod is the standard schema validation library. The Vercel AI SDK's generateObject function provides the TypeScript equivalent of the Instructor pattern: define a Zod schema, pass it to generateObject, and receive a validated typed object.

import { generateObject } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';

const extractionSchema = z.object({
  companyName: z.string().min(1),
  revenueUsd: z.number().positive(),
  fiscalYear: z.number().int().min(2000).max(2030),
  confidence: z.number().min(0).max(1),
});

const { object } = await generateObject({
  model: openai('gpt-4o'),
  schema: extractionSchema,
  prompt: documentText,
});

The generateObject function handles schema generation, structured output configuration, parsing, and basic retry. For more complex retry logic and custom validation error handling, you will need to add that layer above generateObject.

Retry Strategies

Validation failures are not uniformly distributed. Some prompts reliably cause schema violations; others only fail occasionally due to natural output variance. A retry strategy must handle both cases.

Retry with error feedback. The most effective retry strategy includes the validation error in the retry prompt. Instead of simply re-calling the API with the same prompt, include a message like: "Your previous response failed validation with the following error: [error]. Please produce a corrected response." This gives the model the information it needs to correct the specific failure rather than making the same mistake.

Maximum retry count. Two to three retries is the standard upper bound. Beyond three retries, you are likely dealing with a systematic prompt or schema issue that requires debugging rather than retrying.

Exponential backoff on rate limit retries. Retries for validation failures can run immediately (they are not caused by provider load). Retries for rate limit errors (HTTP 429) should use exponential backoff with jitter.

Partial output handling. For streaming responses where the client begins processing before the full response is received, partial JSON is a real case. If using streaming for structured output (e.g., for UX reasons like showing partial results progressively), libraries like partial-json can parse partial JSON into a best-effort object. Mark partial outputs as provisional and do not act on them until the full response is received and validated.

Output Parsing Pipelines

For production systems with complex output requirements, a structured parsing pipeline is cleaner than ad-hoc validation scattered across the codebase.

A well-structured pipeline has four stages: (1) raw output extraction — extracting the JSON or structured content from the model response, handling any prefix or suffix text the model added; (2) syntax validation — parsing as JSON and catching parse errors; (3) schema validation — validating the parsed object against the schema; (4) business logic validation — running domain-specific constraints.

Each stage has its own error handling and logging. Failures at stage 1 or 2 suggest a prompt issue. Failures at stage 3 suggest a schema or provider configuration issue. Failures at stage 4 suggest a business logic constraint that the model is not respecting and may require prompt engineering to address.

Logging failures with full context — the raw model output, the prompt, the model version, the specific validation error — is essential for diagnosing systematic issues. A validation failure that occurs on 0.5% of calls sounds minor until the system is handling 10,000 calls per day, at which point it is 50 validation failures per day that require attention.

Null Safety and Optional Field Handling

Null values in LLM output are a persistent source of silent failures. When a model cannot extract a requested field from the source text — because the information is absent, ambiguous, or expressed in a way the model does not recognise — it may return null, an empty string, a placeholder string like "N/A" or "not specified", or simply omit the field entirely. Each of these behaviours requires different handling, and conflating them leads to downstream bugs that are difficult to trace.

Distinguish between structured nulls and extraction failures. A null returned because the information genuinely does not exist in the source document is semantically different from a null returned because the extraction failed. Design your schema to capture this distinction explicitly. An optional field with Optional[str] = None in Pydantic tells you the field was not populated, but does not tell you whether the source document lacked the information or whether extraction failed. Adding a companion confidence: Optional[float] or extraction_status: Literal['extracted', 'not_present', 'uncertain'] field gives downstream consumers the signal they need to decide how to handle the null.

Preventing silent null propagation. The most dangerous nulls are those that silently flow through the pipeline and corrupt downstream data. A null stored as a database record, used in a calculation, or displayed in a UI without detection can cause incorrect business decisions. Implement null detection at the output parsing stage: after schema validation, check whether the count of null required fields exceeds a threshold (e.g., more than one null in a five-field schema). Log the specific fields that are null with the source document ID. For critical fields where null is unacceptable, treat a null value as a validation failure and route to the retry path or human review queue rather than storing it.

Default values vs. Optional fields. Pydantic allows setting default values for fields that may not be populated. Using a typed sentinel default (None for Optional fields, [] for list fields that may be empty) is preferable to using empty strings or "N/A" strings as defaults, because typed nulls are distinguishable programmatically while string defaults may be indistinguishable from extracted empty strings. In TypeScript with Zod, .optional() produces undefined rather than null, which is important to handle explicitly at the consuming layer.

Handling model refusals for optional content. Some models will produce verbose explanations in a field rather than null when they cannot extract the requested information — "The document does not contain information about the contract value" instead of null. This passes schema validation if the field is typed as string, but the consuming code receives an explanation string rather than the expected value. Add a post-validation cleanup step that detects common refusal patterns (strings containing "not found", "not specified", "the document does not") and normalises them to null before the value is used downstream.

Schema Versioning Across Model Updates

LLM providers update models, and model updates can change output behaviour. An output schema that works reliably with one model version may have higher violation rates after a silent model update. Schema versioning protects against this.

Version your output schemas explicitly. Tag each schema version with a version identifier. Include the schema version in your logging alongside the model version, so you can correlate schema violation rates with specific model versions.

Maintain backward-compatible schema changes. Adding optional fields to an output schema is safe — existing parsing code will ignore unknown fields (in most JSON parsers) or handle them as optional. Removing required fields, changing field types, or renaming fields are breaking changes that require a migration strategy.

Test schemas against multiple model versions. When a provider releases a new model version, run your validation test suite against the new version before migrating. If the violation rate has increased, investigate whether a prompt change or schema adjustment restores the previous rate before rolling out.