Embedding Models and Retrieval System Design

The embedding model is the lens through which your RAG system sees documents. A well-chosen embedding model creates a vector space where the distance between a query and a relevant document is small, and the distance to irrelevant documents is large. A poorly chosen model — one misaligned with your domain or query distribution — creates a vector space where retrieval is essentially arbitrary for your actual use cases, regardless of how sophisticated your chunking or reranking strategy is.

Most engineers treat the embedding model as a configuration choice made at the start of the project and never revisited. This lesson makes the case that embedding model selection is an empirical question that must be answered with evaluation data, not intuition, and that retrieval quality has a measurable, direct effect on end-to-end RAG system quality.

How Embeddings Encode Meaning

An embedding model encodes a piece of text into a dense vector — a list of floating-point numbers — such that semantically similar texts have vectors with high cosine similarity, and dissimilar texts have vectors with low similarity. The model is trained on large corpora with contrastive objectives: given a query and a relevant passage, the model is trained to produce vectors with high similarity; given a query and an irrelevant passage, it is trained to produce vectors with low similarity.

This training objective means the model learns to compress semantic meaning into the vector space. "Machine learning model inference" and "running a neural network in production" may have very different surface forms but will cluster together in a well-trained embedding space. Conversely, "bank" (financial institution) and "bank" (river bank) will ideally map to different regions of the space based on surrounding context — though this disambiguation depends on the quality of the model's contextual understanding.

Dimensionality and its tradeoffs. Higher-dimensional embeddings can encode finer-grained semantic distinctions, but they come with increased storage cost (a 3072-dimension float32 vector requires approximately 12KB per chunk) and increased vector search latency. Models like text-embedding-3-large support Matryoshka Representation Learning (MRL), which allows you to truncate the embedding to fewer dimensions (e.g., 256 or 512) with graceful performance degradation. This gives you a latency-quality dial: use full-dimension vectors for high-precision retrieval, truncated vectors for high-throughput or storage-constrained applications.

How embedding models differ. General-purpose models are trained on broad internet text, which makes them strong on everyday language but potentially weak on specialised domains with distinct vocabulary. Domain-specific models are fine-tuned on domain text to shift the embedding space toward domain-relevant semantic clusters. For highly technical domains — medical literature, legal contracts, scientific papers, source code — the performance gap between a general model and a domain-specific one can be substantial enough to make the difference between a useful RAG system and a useless one.

Embedding Model Selection Criteria

Choosing an embedding model requires evaluating five dimensions specific to your use case.

Task type. Embedding models are often specialised for retrieval (asymmetric: short query vs. long document), semantic similarity (symmetric: comparing two texts of similar length), or clustering (grouping documents by topic). For RAG, retrieval-optimised models are the right category. The MTEB benchmark separates performance by task type; the "Retrieval" category score is the relevant number.

Domain alignment. Compare the domain of your corpus to the training distribution of candidate models. A model trained primarily on web text may embed medical literature poorly because clinical terminology does not appear frequently in web text and therefore does not benefit from the contrastive training signal. The voyage-medical-1 and BioLORD models are purpose-built for biomedical text. Legal-BERT-derived embeddings outperform general models on contract and case law text. For general professional content (business documents, technical documentation, news), general-purpose models perform adequately.

Context length. Some embedding models accept up to 8192 tokens (nomic-embed-text-v1.5, e5-mistral-7b-instruct), which enables longer chunk sizes without truncation. For corpora with long, cohesive documents — academic papers, legal briefs, technical specifications — a long-context embedding model may allow you to embed entire sections rather than fixed-size chunks, improving coherence.

Inference cost and latency. If you are re-embedding at query time (which is always the case for queries), embedding latency contributes directly to end-to-end request latency. Larger models have higher inference latency. text-embedding-3-small can produce an embedding in under 5ms for short queries via the OpenAI API. Self-hosted larger models may take 50 to 200ms under load. Benchmark embedding latency under your expected concurrency before committing to a model.

Fine-tuning feasibility. If your domain is highly specialised and no suitable pre-trained model exists, fine-tuning is an option. Fine-tuning an embedding model requires a dataset of (query, positive document, negative document) triples and a contrastive loss function (typically InfoNCE). The Sentence Transformers library provides the training infrastructure. The data requirement is significant: generally 1,000 to 10,000 high-quality training triples to produce a meaningful improvement.

Fine-Tuning Embedding Models

Fine-tuning an embedding model is justified when: your domain is genuinely specialised with domain-specific vocabulary and relationships, you have evaluated general-purpose models and found retrieval quality insufficient against your evaluation harness, and you can collect or generate the training data.

Training data generation. Generating (query, relevant document) pairs synthetically using an LLM is the most practical approach for most teams. Given a document chunk, ask an LLM to generate three to five questions that the chunk answers. The document chunk is the positive example for each generated question. For negative examples, randomly sampled documents from the corpus work reasonably well; hard negatives (documents that are topically related but do not answer the specific question) produce stronger fine-tuning signal.

Fine-tuning approach. The standard approach is to fine-tune on top of a strong pre-trained model (e.g., bge-large-en-v1.5) rather than training from scratch. This requires much less data and produces a model that inherits the pre-trained model's general semantic understanding while adapting to the domain. Using LoRA adapters for efficient fine-tuning reduces GPU memory requirements significantly and allows fine-tuning on a single A100 or equivalent GPU.

Evaluation after fine-tuning. Always evaluate the fine-tuned model against the baseline using your retrieval evaluation harness before deploying. Fine-tuning can cause catastrophic forgetting on general semantic understanding, degrading performance on query types that were not well-represented in the fine-tuning data. The evaluation harness catches this before it reaches production.

Multi-Vector Retrieval and ColBERT

Standard bi-encoder retrieval encodes an entire document chunk into a single vector. The problem is information compression: a 512-token chunk is compressed into a single point in the vector space, and the model must make tradeoffs about which aspects of the content to preserve and which to sacrifice. A query that aligns with a minority aspect of the chunk may miss it despite the chunk being overall relevant.

ColBERT (Contextualised Late Interaction over BERT) addresses this by storing one vector per token rather than one vector per document. At query time, the query is also encoded into one vector per token. The relevance score between a query and a document is the sum of maximum similarities between each query token vector and the closest document token vector — the MaxSim operation. This late interaction captures fine-grained token-level relationships rather than compressing the entire document into a single embedding.

The practical implications: ColBERT significantly outperforms bi-encoder retrieval on queries where the relevant information is contained in a specific phrase or sentence within a longer chunk, rather than being the dominant topic of the chunk. The tradeoff is storage: storing one vector per token for a large corpus is 50 to 100 times more storage than single-vector bi-encoder indexing. For corpora where retrieval precision is critical and storage is not a constraint, ColBERT is worth evaluating.

The Ragatouille library provides a practical implementation of ColBERT that integrates with LangChain and LlamaIndex. PLAID is an efficient ColBERT indexing algorithm that reduces the storage overhead by clustering token-level vectors.

Sparse-Dense Fusion

Sparse vectors are high-dimensional vectors (often 30,000+ dimensions) where most values are zero, and non-zero values represent the importance of specific vocabulary terms. The SPLADE model generates sparse vectors that encode BM25-like term importance with learned weighting, producing better representations than raw BM25 for retrieval tasks.

Sparse-dense fusion combines SPLADE sparse vectors with dense embedding vectors in a unified retrieval index. This is distinct from running BM25 and vector search in separate indexes and fusing rankings: both sparse and dense signals are encoded together and retrieved in a single query. Pinecone, Qdrant, and Weaviate all support this natively.

The advantage over separate BM25 and vector hybrid search is that SPLADE sparse vectors are learned representations that encode term importance in context, not raw term frequency. They handle morphological variants (run/running/runs mapping to similar sparse vectors) and learn which terms are diagnostically important for retrieval on the training corpus.

Building a Retrieval Evaluation Harness

A retrieval evaluation harness measures how well your retrieval system returns relevant documents before the LLM sees them. End-to-end evaluation (judging the quality of the final LLM answer) conflates retrieval quality with generation quality — a good answer can come from a generation model compensating for poor retrieval, and a bad answer can come from poor generation quality despite perfect retrieval. Measuring retrieval separately gives you a direct signal on the retrieval component.

Building the evaluation dataset. The evaluation dataset consists of (query, list of relevant document IDs) pairs. For 50 to 200 queries, manually identify which chunks in your corpus are relevant. This is the ground truth. The quality of this annotation directly determines the validity of the evaluation — inaccurate or incomplete annotation produces misleading metrics.

MRR (Mean Reciprocal Rank). For each query, MRR assigns a score of 1/rank to the first relevant document retrieved, where rank is its position in the result list. If the first relevant document appears at position 1, the score is 1.0; at position 3, it is 0.33; if not in the top-k, it is 0. MRR averaged across all queries produces the mean reciprocal rank. MRR is most appropriate when you only care about the first relevant result.

NDCG (Normalised Discounted Cumulative Gain). NDCG accounts for the position of multiple relevant documents and their relevance grades. It rewards systems that rank highly relevant documents higher and discounts the value of relevant documents appearing lower in the results. NDCG@k where k equals the number of chunks passed to the LLM is the standard metric for RAG retrieval evaluation — not a fixed NDCG@10, which may not reflect your actual context window budget.

Contextual compression. After retrieval, contextual compression removes sentences or passages from retrieved chunks that are not relevant to the specific query. The LLMChainExtractor in LangChain does this by asking an LLM to extract only the relevant portions of each retrieved chunk. This reduces the amount of irrelevant content in the LLM context, improving answer quality and reducing token costs. The tradeoff is additional LLM latency and cost for the compression step.