RAG Architecture and Design

By the end of this lesson, you can take a Claude- or GPT-4-based RAG feature that confidently returns wrong answers and trace the failure to a specific decision in your chunking, embedding, or retrieval stack — instead of assuming the model itself is the problem.

Retrieval-Augmented Generation is the most widely deployed LLM architecture pattern in production as of mid-2026. The concept is straightforward: before calling the LLM, retrieve documents relevant to the query from an external store and inject them into the context. The LLM reasons over real data rather than relying on training-time knowledge alone.

The implementation is where most teams discover that "retrieval" is not a single decision but a stack of decisions, each of which has significant quality and cost implications. A RAG system that retrieves the wrong chunks, embeds with a mismatched model, or runs pure vector search on keyword-heavy queries can perform worse than a simple keyword search system. Getting retrieval right is the single highest-leverage investment in a RAG system's quality.

Why Chunking Strategy Determines Retrieval Quality

Documents do not arrive in embedding-ready form. Before a document can be retrieved, it must be split into chunks — segments that are individually embedded and stored in the vector index. The chunking strategy determines what information can and cannot be retrieved.

Fixed-size chunking splits documents by token count, typically 256 to 1024 tokens, with an optional overlap between consecutive chunks. It is simple to implement, predictable in storage cost, and works reasonably well for homogeneous text. The failure mode is semantic fragmentation: a sentence that begins at the end of one chunk and continues at the start of the next is split across two embeddings. Neither chunk accurately represents the complete thought, so retrieval may miss both.

The chunk overlap parameter directly addresses this: an overlap of 10 to 20% of the chunk size means that content near chunk boundaries appears in two consecutive chunks, reducing the probability of a critical sentence being split. For most production use cases where structural chunking is not feasible, a 512-token chunk with 100-token overlap is a defensible starting point.

Semantic chunking groups text by semantic boundary rather than token count. The implementation embeds every sentence (or short sentence group), then computes cosine similarity between consecutive embeddings. Where similarity drops sharply, a new chunk boundary is placed. This produces chunks that correspond to coherent topics rather than arbitrary token windows, which generally improves retrieval precision for informational queries.

The practical limitation is cost: semantic chunking requires an embedding call per sentence during ingestion, which can be 50 to 100 times more expensive than fixed-size chunking for large corpora. It is most justified for curated knowledge bases where ingestion happens once and retrieval quality is the dominant concern.

Structural chunking uses document structure — headers, sections, paragraphs, tables, list items — as the natural chunking boundary. For well-structured documents (technical documentation, contracts, policies, code files), structural chunking is almost always superior to fixed-size chunking because the document's own authors have already identified semantic boundaries.

For PDF-based corpora, structural chunking requires reliable structure extraction. PDFs that were scanned rather than generated digitally require OCR before structure can be detected. Azure Document Intelligence and AWS Textract both provide layout-aware extraction that identifies headers, tables, and paragraphs, enabling structural chunking from complex PDF layouts.

Chunk size and its effect on retrieval precision vs. recall. Smaller chunks embed more specifically, improving retrieval precision for specific factual queries — but they lose surrounding context, so the retrieved chunk may not contain enough information for the LLM to answer. Larger chunks preserve context but embed more diffusely, reducing retrieval precision for specific queries. A common production pattern is parent-child chunking: store small child chunks for high-precision retrieval, but when a child chunk is retrieved, return its parent chunk (or a larger context window around it) to the LLM. This combines fine-grained retrieval with sufficient context for generation.

Embedding Model Selection

The embedding model converts text into a vector representation. The quality of that representation determines how well the vector index can distinguish relevant from irrelevant documents. Embedding model selection has a direct, measurable effect on retrieval quality.

OpenAI text-embedding-3-small is the default choice for most production systems. At 1536 dimensions and a very low per-token cost (approximately $0.02 per million tokens as of mid-2026), it handles general English retrieval well. The tradeoff is dimension count relative to text-embedding-3-large and ceiling performance on complex semantic relationships.

OpenAI text-embedding-3-large produces 3072-dimensional embeddings and scores significantly higher on MTEB (Massive Text Embedding Benchmark) retrieval tasks. For corpora where retrieval precision is the primary concern and you are willing to pay 5x the embedding cost and accept higher vector storage costs, 3-large is the better choice. The performance difference is most visible on long-tail, semantically complex queries rather than simple factual lookups.

Open-source alternatives — particularly the bge-large-en-v1.5 family from BAAI, nomic-embed-text-v1.5, and the E5 family — have closed much of the gap with proprietary models. The primary advantage is data privacy: open-source embedding models can be self-hosted, so documents never leave your infrastructure. For healthcare, legal, and financial use cases with strict data residency requirements, this is often the deciding factor.

Domain-specific embedding models matter when your corpus is far from the distribution that general embedding models were trained on. Medical terminology, legal citations, code, and multilingual content all benefit from domain-adapted embeddings. For code retrieval specifically, models like voyage-code-2 significantly outperform general text embeddings.

The crucial practice: evaluate on your data. MTEB scores are measured on standardised benchmarks that may not reflect performance on your specific corpus and queries. Build a retrieval evaluation dataset — 50 to 200 representative queries with known relevant documents — before selecting an embedding model. Run all candidate models against it and measure recall@k (what fraction of relevant documents appear in the top-k retrieved results). This takes a day of engineering time and can prevent choosing an embedding model that performs well on benchmarks but poorly on your domain.

Vector Database Selection

The vector database stores embeddings and handles approximate nearest-neighbor search at query time. The selection criteria are: hosting model (managed vs. self-hosted), metadata filtering capability, scale requirements, consistency model, and operational overhead.

Pinecone is a fully managed vector database with a serverless tier. It requires zero infrastructure management, scales automatically, and supports metadata filtering on indexed fields. The tradeoff is vendor lock-in and cost at high query volumes. Pinecone is the right choice for teams that want to ship a RAG feature without owning vector database operations.

Weaviate is open-source and can be self-hosted or used as Weaviate Cloud. It offers built-in hybrid search (BM25 and vector in a single query), a GraphQL API, and object-level access control. The module system allows adding cross-encoder reranking and multi-modal retrieval directly within Weaviate. Self-hosted Weaviate gives full data control but requires Kubernetes-level operational capability.

pgvector is a PostgreSQL extension that adds vector similarity search to an existing Postgres database. If you already run Postgres in production, pgvector is the lowest-friction path to vector search: it uses your existing database infrastructure, supports full SQL-level metadata filtering, and participates in ACID transactions alongside your relational data. The limitation is scale: pgvector's HNSW and IVFFlat indexes begin to degrade in query latency above approximately 5 million vectors on most Postgres hosting configurations. For RAG systems with small to medium corpora, pgvector is often the right answer.

Qdrant is open-source and self-hostable, with a managed cloud option. It offers sparse-dense hybrid search natively, supports payload-based filtering with high selectivity, and has strong multi-tenancy support for multi-user RAG systems where each user's data must be isolated. Qdrant is particularly well-suited for systems where filtering precision is critical — for example, a knowledge base where each document is tagged to a department and users should only retrieve documents from their department.

The right choice depends on your constraints: Pinecone for zero-ops managed hosting, pgvector if Postgres is already in your stack and the corpus is under 5 million vectors, Qdrant for multi-tenancy and fine-grained filter performance, and Weaviate when you want native hybrid search and rich schema capabilities in a single service.

Hybrid Search: Combining BM25 and Vector Retrieval

Pure vector search has a well-known failure mode: keyword specificity. If a user asks about "ISO 27001 certification renewal" and the document corpus contains a section with that exact title, the section may not score highly in vector space if the embeddings collapse the specific term into a general concept cluster. BM25 keyword search, by contrast, rewards exact term overlap and handles rare terms, product names, and identifiers well.

Hybrid search combines the two signals. The standard implementation runs BM25 and vector search in parallel, then fuses the result sets using Reciprocal Rank Fusion (RRF). RRF combines rankings from multiple retrieval systems without requiring score normalisation: each document's fused score is the sum of 1/(k + rank_i) across all retrieval systems, where k is a smoothing constant (typically 60). The fused ranking consistently outperforms either system alone on mixed query types.

Most production RAG systems should default to hybrid search rather than pure vector search unless the query type is so homogeneously semantic that keyword overlap is irrelevant.

Reranking with Cross-Encoders

Vector search and BM25 are both bi-encoder approaches: the query and each document are embedded independently, and similarity is computed by dot product or cosine distance. This is efficient — one embedding per query, one per document at index time — but the independence means the model cannot attend to query-document interactions during ranking.

Cross-encoders take the query and a candidate document together as a single input and produce a relevance score. They can attend to the full relationship between query and document, producing significantly higher-quality rankings. The tradeoff is latency: a cross-encoder must score each candidate separately, making it 10 to 100 times slower than bi-encoder retrieval.

The standard production pattern is a two-stage pipeline: retrieve the top-50 or top-100 candidates using vector or hybrid search, then rerank those candidates using a cross-encoder, and pass the top-5 or top-10 reranked results to the LLM. The reranker adds 50 to 150ms of latency depending on the candidate set size and model, which is acceptable for most interactive applications.

Cohere Rerank is the most widely used managed cross-encoder reranking service. Open-source cross-encoders from the cross-encoder/ms-marco-MiniLM family can be self-hosted for lower cost and data privacy. ColBERT's late interaction mechanism is a middle ground — it achieves cross-encoder-quality ranking with lower inference cost by pre-computing token-level representations.

The diagram below traces the full path a query takes through this stack, end to end: BM25 and vector search run in parallel, their results are fused with RRF, the fused candidate set is reranked by a cross-encoder, and only the reranked top results are passed into the LLM's context window.

Metadata Filtering in Practice

Metadata filtering restricts the search space before or during vector search, ensuring that retrieved documents satisfy structural constraints. A multi-tenant knowledge base where users should only retrieve their own organisation's documents, a versioned documentation system where queries should only retrieve the current version, and a time-limited news summariser where results should be restricted to the past 30 days all require metadata filtering.

The key engineering decision is whether filtering happens pre-retrieval (filter first, then search the filtered set) or post-retrieval (search all vectors, then filter results). Pre-filtering produces smaller candidate sets and deterministic result counts, but requires the vector index to support efficient payload-based filtering without full scans. Qdrant's HNSW implementation with payload filtering is particularly strong here: it indexes payload fields alongside vectors and can apply filters during graph traversal rather than post-hoc. pgvector with a WHERE clause achieves the same result for SQL-compatible filter expressions.

Post-filtering is simpler to implement but can produce fewer results than the requested top-k if many retrieved documents fail the filter, degrading retrieval quality unpredictably.