How LLMs Work: An Engineering Mental Model

Before you can make good engineering decisions about AI-powered systems, you need an accurate model of what is happening when you call an LLM API. Not a deep mathematical treatment of transformer architectures, but a precise enough understanding that you can reason about performance, cost, failure modes, and the limits of what prompting alone can solve.

This lesson covers what software engineers need to know about how LLMs work, with direct application to the decisions you face when building production systems.

Tokens, Context Windows, and Production Context Management

LLMs do not process text character by character or word by word. They process tokens: roughly 0.75 words in English on average, though this varies significantly by language, code, and special characters. When you call the OpenAI API with a 1,000-word prompt, you are sending roughly 1,300 tokens. The model's response adds more tokens. Everything costs per token, in money and in latency.

The context window is the total number of tokens the model can process in a single inference call: both input and output combined. GPT-4o has a 128,000 token context window. Claude 3.7 Sonnet supports up to 200,000 tokens. These numbers sound large. In production systems, they fill up faster than you expect.

Consider a customer support application where each conversation history is appended to every new request. After 20 exchanges, the accumulated conversation plus system prompt may consume 8,000 to 12,000 tokens before the user's latest message is even added. If the system also injects retrieved documents or tool outputs, the context window fills quickly. What happens when it fills?

The model truncates. Different providers handle truncation differently: some truncate from the beginning of the context (dropping the oldest messages), some return an error. Either way, critical context gets silently dropped. In a support context, that means the model loses the history of what the user has already tried, leading to repeated suggestions and frustrated users.

Context management is an engineering design problem. Approaches include: conversation summarization (periodically compress older turns into a summary), sliding window (keep only the most recent N turns), selective retrieval (store conversation history in a vector database and retrieve only the most relevant segments), and explicit context budget allocation (reserve token budgets for each content type and enforce them before the API call).

Training vs. Inference: What the Model Knows and What It Retrieves

An LLM learns during training by processing billions of tokens of text and adjusting billions of numerical weights to minimize prediction error. When training ends, those weights are frozen. The model cannot learn from new information at inference time.

This has a direct consequence: the model's knowledge has a training cutoff. Ask it about events after that date and it will either say it does not know or, more commonly, hallucinate a plausible-sounding answer based on patterns from pre-cutoff training data. Hallucination is not a bug introduced during deployment. It is a structural property of how the model was built.

Hallucination happens because the model has no verification mechanism. It does not know what it does not know. When asked a question outside its training distribution, it still generates the most statistically likely continuation, which may be a confident fabrication. The model has no internal flag that distinguishes "I know this" from "I am generating something plausible."

This is why retrieval-augmented generation (RAG) matters: you cannot reliably fix hallucination by prompting the model to "only answer questions you know the answer to." You can reduce hallucination in a specific domain by grounding the model's responses in retrieved, authoritative documents.

Temperature and Top-p: What Engineers Need to Know

Temperature and top-p are sampling parameters that control how the model selects each output token from the distribution of possible next tokens.

At temperature 0, the model always selects the highest probability token. Output is deterministic (for the same model and input). At temperature 1.0, the model samples according to the raw probability distribution. Higher temperatures increase randomness and creative variation. Lower temperatures increase consistency and reduce unexpected outputs.

Top-p (nucleus sampling) works differently: instead of scaling the full distribution by temperature, it restricts sampling to the smallest set of tokens whose cumulative probability exceeds the threshold value. At top-p 0.9, the model only considers tokens in the top 90% of the probability mass. This trims the long tail of low-probability tokens.

When do engineers need to care about these settings? For structured output tasks, where the model must produce JSON, code, or formatted data, lower temperature (0 to 0.3) reduces the probability of format deviations. For creative generation tasks, higher temperature produces more varied output. For production systems where consistency matters more than variety, lower temperature reduces the variance in outputs.

Most LLM API calls do not require careful parameter tuning. The default settings are appropriate for the majority of use cases. The cases where they matter are: structured output generation, tasks with high consistency requirements, and cases where you are seeing unexpectedly variable outputs in testing.

RAG, Fine-Tuning, and Prompting: The Engineering Decision

Three primary approaches exist for adapting LLM behaviour to a specific use case. Choosing between them is an engineering decision with significant cost, complexity, and performance implications.

Prompting modifies the model's behaviour by providing instructions, examples, and context in the input. It requires no model changes, is fast to iterate, and works well when the task can be fully specified in a context window. Its limits: it cannot inject information that exceeds the context window, it does not persist across calls, and it cannot teach the model new factual knowledge.

Retrieval-augmented generation (RAG) retrieves relevant documents from an external store at query time and injects them into the context. It solves the knowledge cutoff problem for specific domains, allows the model to reference proprietary or current information, and scales to large knowledge bases. Its limits: retrieval quality depends on chunking and embedding quality, retrieved context consumes token budget, and poorly chunked documents can confuse the model rather than help it.

Fine-tuning trains the model on domain-specific examples to adjust its weights. It is appropriate when the task requires a style, format, or capability that cannot be achieved through prompting alone, when you need to reduce prompt length by baking in instructions, or when you need consistent response patterns across a very high volume of calls. It is expensive to run, slow to iterate, and requires a labelled dataset of examples. For most production use cases, exhausting prompting and RAG options before committing to fine-tuning is the right engineering order.

Latency, Cost, and Throughput in Production

LLM API calls are slow and expensive relative to conventional API calls. A single GPT-4o call with a 2,000 token prompt and 500 token response may take 3 to 8 seconds and cost $0.02 to $0.05 depending on model tier and provider. At scale, these numbers matter.

Latency in LLM systems comes from two sources: time to first token (the delay before any output starts streaming) and time to last token (the delay until the full response is complete). For user-facing features, streaming responses to the client as they generate significantly improves perceived performance, even when total generation time is unchanged.

Cost scales with token volume. The primary cost levers are: model tier selection (GPT-4o is roughly 5x the cost of GPT-4o mini per token), prompt length reduction, output length constraints (instruct the model to be concise when brevity is acceptable), and caching (some providers offer prompt caching for frequently repeated system prompts).

Throughput is constrained by rate limits at the API level. All major LLM providers impose requests per minute and tokens per minute limits. For batch processing workloads, these limits require queue management, backoff logic, and throughput planning that does not apply to conventional API integration.