Multimodal Systems and Advanced Integration Patterns — Building Production LLM Systems

What you'll learn

Integrate vision models (GPT-4o, Claude 3.7, Gemini 2.5 Pro) into a document processing pipeline and select the appropriate model for described image analysis tasks
Design a PDF-to-structured-data pipeline using layout-aware document intelligence platforms and explain where LLM generation fits vs. where deterministic extraction is more appropriate
Evaluate when multimodal adds genuine value vs. when it adds complexity without proportional quality benefit for a described use case
Implement a multimodal RAG system that combines text and image embeddings for a described retrieval scenario

The LLM ecosystem has converged on multimodal as the default. GPT-4o, Claude 3.7 Sonnet, and Gemini 2.5 Pro all accept text, images, documents, and audio as inputs, and the performance gap between vision-capable models and text-only models has largely closed. In mid-2026, most production LLM features could be multimodal if the use case warrants it.

The engineering question is not whether multimodality is available but whether it adds value that justifies the complexity and cost. Vision models cost more per call than text-only models, multimodal pipelines are harder to evaluate, and many use cases that appear to require vision can be served just as well with text extraction alone. This lesson covers when to reach for multimodal, how to build reliable multimodal pipelines, and where the complexity traps are.

Vision Model Capabilities and Selection

GPT-4o handles images up to 20MB, processes multiple images per request, and supports vision in the same API interface as text. For most general-purpose visual understanding tasks — document screenshots, diagrams, charts, UI mockups, natural scenes — GPT-4o provides strong performance with reasonable latency. The detail parameter controls how the image is processed: "high" tiles large images for fine-grained analysis (costs more tokens) and "low" provides a low-resolution overview.

Claude 3.7 Sonnet (vision) is particularly strong at dense document understanding — extracting structured information from complex layouts, tables, and forms. For document intelligence use cases where the document structure is as important as the text content, Claude 3.7's spatial reasoning over document layouts is a meaningful differentiator. Claude accepts images up to 3.75MB and up to 20 images per request.

Gemini 2.5 Pro has a significantly longer context window (2M tokens) and native video and audio input capabilities. For use cases that require processing long documents (hundreds of pages), multi-frame video analysis, or audio alongside visual content, Gemini 2.5 Pro's extended context and native multimodal support provide capabilities that GPT-4o and Claude 3.7 do not match. The tradeoff is latency: Gemini 2.5 Pro's large context capability comes with higher inference latency on complex inputs.

Model selection by task type:

General document screenshots and diagrams: GPT-4o
Complex table, form, and contract layout extraction: Claude 3.7 Sonnet
Long documents (50+ pages), video analysis, audio: Gemini 2.5 Pro
Native image generation (not covered here): DALL-E 3, Stable Diffusion XL, Flux

Document Processing Pipelines

The most common production use case for vision models is document processing: converting PDFs, scanned images, or complex document layouts into structured data. The pipeline design depends on the document type and the required output.

Layer 1: Layout detection. Before passing a document to a vision model, extract the structure. Azure Document Intelligence and AWS Textract provide layout-aware OCR that identifies document structure elements: paragraphs, tables, headers, form fields, key-value pairs, and their bounding boxes. This is significantly cheaper than passing the full document as an image to a frontier vision model and produces more reliable structure extraction for standard document types.

Layer 2: Selective vision model usage. Once structure is extracted, use vision models selectively: for content that the layout extraction cannot handle reliably (complex hand-drawn diagrams, degraded scans, unusual layouts), and for semantic understanding that requires reasoning about the visual layout alongside the text content. Do not default to sending every document page as an image to a vision model — the cost and latency penalty is substantial for high-volume document pipelines.

The hybrid approach: For a contract review pipeline, Azure Document Intelligence extracts paragraphs, table structure, and key-value pairs deterministically. Clause identification is done by a text LLM working on the extracted text. Complex tables with merged cells or unusual formatting that break the layout extraction are sent as image crops to Claude 3.7 for interpretation. This hybrid approach reduces vision model calls to 5-15% of documents while maintaining coverage on the hard cases.

Warning

Vision model token costs are substantially higher than text token costs. A high-detail image of a dense document page can consume 1,500 to 3,000 tokens just for the image encoding, before any text in the prompt. For a document processing pipeline handling thousands of pages per day, defaulting to vision model processing for every page will produce unexpectedly high costs. Always calculate the per-document token cost for your image inputs before deploying a vision pipeline at scale.

Audio and Speech Integration

Whisper (OpenAI) is the standard model for speech-to-text transcription. The Whisper API accepts audio files up to 25MB in mp3, mp4, wav, and other common formats and produces text transcripts with optional word-level timestamps. For real-time transcription, the Whisper model weights are available for self-hosting, which provides lower latency than the API for streaming audio.

Real-time STT (Speech-to-Text). For real-time applications (voice agents, live transcription), the OpenAI Realtime API and Google's Speech-to-Text streaming API both support WebSocket-based streaming transcription. Latency characteristics differ significantly: the Realtime API's transcription operates at 200-500ms word-level latency; traditional HTTP-based Whisper transcription requires buffering audio segments, typically 2-5 seconds of audio per API call.

Audio in the LLM context. Audio content can be included directly in Gemini 2.5 Pro requests (up to 9.5 hours of audio per request via the Files API). For most production use cases, the practical pattern is: transcribe audio to text using Whisper or similar, then pass the transcript to a text LLM. Direct audio input to an LLM is most useful when audio characteristics (tone, pacing, speaker separation) are as important as the verbal content.

Multimodal RAG

Standard RAG retrieves text documents. Multimodal RAG retrieves both text documents and images (or documents containing images), and passes the retrieved images alongside retrieved text to a vision LLM for generation.

Image embedding. Images can be embedded into a shared vector space with text using multimodal embedding models: CLIP and its successors (SigLIP, EVA-CLIP) embed images and text into a single space where semantically related images and text queries have high cosine similarity. This enables cross-modal retrieval: a text query "quarterly revenue chart" retrieves relevant chart images from the corpus based on semantic alignment between the query text and the chart's visual content.

When multimodal RAG adds value. Multimodal RAG is genuinely superior to text-only RAG when: the corpus contains diagrams, charts, or figures that are referenced by text but whose key information is in the visual content rather than captions; the corpus contains scanned documents where the visual layout carries meaning that OCR cannot capture; or the user's query requires visual understanding ("what does the assembly diagram for step 3 look like?"). For purely text-based knowledge bases with machine-generated PDFs, multimodal RAG adds complexity without benefit.

ColPali for document image retrieval. Standard multimodal RAG with CLIP embeddings works well for natural images but has a specific weakness for document page retrieval: CLIP was trained on image-caption pairs and does not represent the fine-grained text content of document pages well. ColPali (Contextual Late-interaction over PaliGemma) addresses this by using a vision-language model (PaliGemma) to produce patch-level embeddings from document page images — one vector per visual patch rather than one vector per page. The MaxSim retrieval mechanism, similar to ColBERT's token-level scoring, computes relevance by finding the best patch-level match between query and document. For document corpora where text on the page is as important as visual layout, ColPali significantly outperforms CLIP on retrieval precision, at the cost of higher storage (patch-level vectors rather than a single image vector per page).

Practical multimodal RAG pipeline. Index documents at two levels: extract text for text embeddings (for text-based retrieval) and store page-level image renders for visual queries. At retrieval time, run both text and image embedding queries against the respective indexes, fuse results using RRF, and pass the top-k results (both text chunks and image crops) to a vision LLM for generation. Implement fallback: if no relevant images are retrieved, fall back to text-only generation rather than passing empty image slots.

Document Intelligence Platforms

Azure Document Intelligence (formerly Form Recognizer) and AWS Textract are the two dominant managed document intelligence platforms as of mid-2026.

Azure Document Intelligence offers pre-built models for common document types (invoices, receipts, identity documents, tax forms, contracts) and a custom model training workflow for proprietary document types. The layout model extracts full structural information from arbitrary document layouts. The key advantage over raw OCR is the structured output: fields are returned as typed key-value pairs rather than raw text, tables are returned with row and column structure, and the bounding box coordinates enable verification UIs.

AWS Textract is deeply integrated with the AWS ecosystem (S3, Lambda, Step Functions) and provides similar capabilities: layout extraction, table extraction, key-value pairs, and query-based extraction (where you specify the fields you want to extract as natural language queries). For teams already on AWS, Textract is the natural choice; for Azure-based teams or multi-cloud setups, Document Intelligence is comparable.

Open-source document extraction alternatives. For teams with data residency requirements or budget constraints that rule out managed cloud services, two open-source libraries have become production-viable in 2025-2026.

Docling (from IBM Research) handles PDF, DOCX, XLSX, and image inputs with a unified output format — DoclingDocument — that captures layout, tables, reading order, and figure references. The key advantage over raw PDF parsing libraries is that Docling uses a layout model (a fine-tuned document layout segmentation model) to identify document structure rather than relying on PDF metadata, which is often unreliable or absent. For digitally generated PDFs, Docling achieves quality comparable to Azure Document Intelligence for structured extraction, with no per-page cost. The tradeoff is compute: the layout model requires GPU inference for high throughput, and Docling is slower than managed cloud services on CPU-only infrastructure.

Marker converts PDFs to clean Markdown by combining a PDF rendering layer, a layout detection model, and an OCR layer for scanned content. The output is Markdown suitable for chunking and embedding, rather than structured key-value pairs. Marker is particularly well-suited for research paper and documentation corpora where the goal is clean text extraction for RAG rather than structured field extraction.

The selection decision: for structured field extraction from standard business document types at scale, Azure Document Intelligence or AWS Textract are more cost-effective (including engineering and maintenance overhead). For pipelines where data cannot leave your infrastructure, or for research/documentation corpora where Markdown output is the goal, Docling and Marker are production-grade alternatives.

Where LLM generation fits. Document intelligence platforms handle structured extraction of known field types well. LLM generation is most valuable for: interpreting ambiguous or non-standard values that the extraction model flags as low-confidence, synthesising extracted data into prose summaries or analyses, and handling document types or layouts not covered by pre-built models.

Building a hybrid document processing pipeline for insurance claims

Backend Engineer

Context

An insurance technology company needed to process incoming claim documents: PDFs containing a mix of structured forms (claim amounts, dates, policy numbers) and unstructured narrative sections (description of the incident). Processing volume was 2,000 documents per day. Initial implementation sent every document page as an image to GPT-4o for extraction, but costs were $3,200 per day and latency was 8-12 seconds per document.

Action

The team redesigned the pipeline as a three-tier system. Tier 1: Azure Document Intelligence pre-built invoice and form models extracted structured fields (claim amount, date of loss, policy number, claimant name) at $0.01 per page with sub-second latency. Tier 2: Extracted text from the narrative sections was passed to gpt-4o-mini for semantic categorisation and incident summarisation — dramatically cheaper than GPT-4o for this text-based task. Tier 3: Only documents where Document Intelligence returned low-confidence extractions (approximately 12% of documents) were routed to GPT-4o vision for manual field verification. The routing logic checked Azure's confidence scores on extracted fields and sent documents below a 0.8 confidence threshold to vision review.

Outcome

Per-document processing cost dropped from $1.60 to $0.22 — an 86% cost reduction. Latency improved to 2-4 seconds for 88% of documents (tier 1 and 2 only) and 6-8 seconds for the 12% routed to vision review. Extraction accuracy on structured fields was maintained at the same level as the pure vision approach, because Document Intelligence's pre-built models performed well on standard insurance form layouts.

Knowledge check

A team is building a knowledge base assistant for a technical product. The knowledge base contains 10,000 articles that are text-only Markdown files, plus 500 technical diagrams stored as PNG files. Users sometimes ask questions that can only be answered by looking at the diagrams (e.g., 'What does the network topology diagram look like?'). The team is evaluating whether to implement multimodal RAG or to have technical writers add text descriptions of each diagram to the articles. Which approach is more appropriate for this use case and why?

Select one answer.

Quick check

Why does this lesson say CLIP embeddings underperform when the task is retrieving pages of a text-heavy document?

Select one answer.

Exercise

Your Task

Design a multimodal document processing pipeline for the following scenario: a construction company receives 300 project documents per day in mixed formats — typed PDFs of specifications, scanned PDFs of hand-signed contracts, and photographs of site conditions. The company needs to extract: structured metadata (project name, contract date, contractor name, total value) from specifications and contracts, incident descriptions from site condition photographs, and a flag indicating whether any document contains hand-written content. Specify: (1) which document types should use Azure Document Intelligence or AWS Textract and which should go directly to a vision LLM, (2) how you would handle the hand-written content detection, (3) the routing logic between processing tiers, (4) how you would estimate and control the per-document cost, and (5) how you would evaluate extraction accuracy.

Your reflection

Did you complete this exercise? What did you find? (Saved locally in your browser)

Key takeaways

Multimodal adds genuine value when visual content cannot be adequately represented in text. For large corpora of photographs, diagrams, scanned documents, or visually-encoded information, multimodal retrieval and generation improve quality. For text-first corpora with machine-generated PDFs, multimodal adds cost and complexity without proportional quality benefit.
Vision model token costs are substantially higher than text token costs — a high-detail document page can consume 1,500 to 3,000 tokens just for the image encoding. Calculate per-document image token cost before scaling any vision pipeline.
Hybrid document processing pipelines — document intelligence platform for structured extraction, text LLM for semantic tasks, vision LLM selectively for low-confidence or non-standard layouts — reduce cost by 70-90% compared to pure vision model approaches for standard document types.
Multimodal RAG combines text and image embedding into a shared vector space using CLIP-family models, enabling cross-modal retrieval where a text query retrieves semantically relevant images. It is most justified when the corpus is large and the visual content cannot be described adequately in text.
For audio-to-LLM pipelines, the practical pattern is: transcribe with Whisper, then pass transcript to a text LLM. Direct audio input to LLMs is most valuable when audio characteristics (tone, speaker identity, pacing) matter alongside verbal content.