Prompt Caching and Cost Optimization at Scale

LLM API costs are not flat and they are not predictable from a simple per-call estimate. They vary by model tier, by token type (input vs. output), by whether the input hits a cache, by the distribution of task complexity across your traffic, and by operational costs that do not appear in the per-token billing at all. Engineers who estimate LLM costs as "price per token times expected traffic" consistently underestimate by 2x to 5x.

This lesson covers the engineering mechanisms that actually control cost at scale: prompt caching, model routing, batching, token budget enforcement, and total cost of ownership modelling. The goal is not to minimise cost in absolute terms — the goal is to deliver the required quality at the minimum cost, which requires understanding what drives cost in the first place.

How Prompt Caching Works

LLM inference has two phases: the prefill phase (processing the input tokens) and the decode phase (generating output tokens). Prefill is computationally expensive because every input token must attend to every other input token in the context. For large prompts — long system prompts, retrieved documents, conversation history — prefill is the dominant cost.

Prompt caching reduces prefill cost by caching the KV (key-value) attention computation for stable prompt prefixes. When a subsequent request has an identical prefix, the cached KV state is reused instead of recomputing it, significantly reducing latency and token cost.

Anthropic prompt caching is opt-in and requires explicitly marking cache breakpoints in the prompt. You add "cache_control": {"type": "ephemeral"} to the content block you want to cache. Content up to and including that breakpoint is cached. Anthropic charges a higher price for the first write to the cache and a significantly lower price (approximately 90% discount on input tokens) for subsequent reads. The cache TTL is 5 minutes by default, with an extended TTL available for longer-lived stable content.

For a RAG system with a large system prompt and retrieved documents, the optimal structure is: stable system prompt with a cache breakpoint, then the retrieved documents (which change per query), then the user query. The stable system prompt is cached after the first request, and all subsequent requests in the cache window pay the cache read price for it.

OpenAI prompt caching is automatic rather than opt-in — there are no cache breakpoints to mark. OpenAI automatically caches prompt prefixes and applies a 50% discount on cache hit input tokens. The cache applies to the first 1024 tokens and then in 128-token increments after that. For OpenAI caching to be effective, prompts must have a stable prefix of at least 1024 tokens that is identical across requests.

Cache-Friendly Prompt Structure

Cache-friendly prompt design is a structural discipline, not a writing discipline. The rule is simple: the more stable a piece of content, the earlier it should appear in the prompt.

Optimal structure for a RAG application:

1. System prompt (most stable — changes with feature releases, not per request)
2. Retrieved documents or reference content (stable within a session, changes across queries)
3. Conversation history (stable within a turn, grows over the session)
4. Current user message (changes every request)

For Anthropic caching, place a cache breakpoint after the system prompt (to cache just the system prompt) and optionally after the retrieved documents (to cache the session context for multi-turn conversations). For OpenAI caching, ensure the system prompt and any stable instructional content appear in the first 1024+ tokens of every request.

Documents vs. instructions. For RAG systems where the same document set is used across multiple queries (e.g., the same product documentation corpus for all users), consider pre-loading the documents into the cache by making a warmup request that includes the full document set before the first user query. This ensures the first user query in a session also benefits from the cache rather than paying the full prefill cost.

Context Window Cost Model

Context window pricing has two components: input tokens and output tokens. Understanding the ratio and distribution of each is essential for accurate cost modelling.

Input tokens include everything in the prompt: the system prompt, retrieved documents, conversation history, and the user message. Input token costs are typically lower than output token costs but dominate the input side because prompts in production RAG systems can be 2,000 to 8,000 tokens per request.

Output tokens are the generated response. Output token generation is more expensive per token than input token processing (roughly 3x to 5x depending on the model). For structured output tasks (short JSON responses), output tokens may be 50 to 200 tokens. For long-form generation tasks (summaries, reports, analysis), output tokens may be 500 to 2,000 tokens per request.

Cost per feature. For each LLM feature in your product, calculate: average input tokens per request (measure from logs, not estimates), average output tokens per request, requests per day at target scale, and the total daily token cost. Do this for each model tier you use. This produces a cost-per-feature-per-day number that you can track and alert on.

Output token dominance at scale. For high-traffic features with substantial generation, output tokens can dominate costs even at a lower per-token price. A feature that generates 1,000 tokens per request at 100,000 requests per day generates 100 million output tokens per day — at $15 per million output tokens, that is $1,500 per day or $45,000 per month for output alone.

Model Routing

Not every LLM task requires the most capable model. A routing strategy assigns each task to the cheapest model that can handle it at the required quality level.

Task classification for routing. Classify your tasks by capability requirements:

• Simple classification and extraction tasks with a well-defined schema: smaller, cheaper models (GPT-4o-mini, Claude Haiku) can handle these at quality comparable to frontier models at 10-20% of the cost.
• Structured data extraction from well-structured inputs: mid-tier models.
• Complex reasoning, multi-document synthesis, long-form generation: frontier models (GPT-4o, Claude 3.7 Sonnet) are required.

Implementing routing. The simplest routing strategy uses a proxy or gateway that intercepts requests and redirects them based on a task type field in the request metadata. LiteLLM supports multi-provider routing with load balancing and fallback. A/B test each routing decision against your eval set before deploying: the quality difference between models is not always where intuition suggests.

Cascading routing. A cascade strategy tries the cheaper model first and escalates to a more expensive model if the output fails validation or quality thresholds. For tasks where most inputs are straightforward but a tail of complex inputs requires a stronger model, cascading can reduce average cost by 40-70% while maintaining quality on hard inputs. The cost of the escalation (double API call) is offset by the savings on the majority of simple cases.

Batching Strategies

For non-interactive workloads — background processing, nightly batch jobs, bulk data enrichment — batching can significantly reduce cost and latency overhead.

OpenAI Batch API. OpenAI offers a Batch API that processes requests with a 24-hour turnaround at 50% of the standard API cost. For workloads that can tolerate next-day processing (content enrichment, weekly summaries, data classification pipelines), the Batch API halves the model cost. Submit requests as JSONL files, poll for completion, and retrieve results in bulk.

Parallelism control. For real-time batch workloads that need faster turnaround than the Batch API allows, control parallelism to maximise throughput within rate limits. Use asyncio (Python) or Promise.all (TypeScript) to send requests concurrently, with a semaphore limiting concurrent requests to just below your provider rate limit. This maximises throughput without triggering rate limit errors.

Token counting before sending. Use provider token counting utilities (tiktoken for OpenAI, Anthropic's token count endpoint) to measure prompt token count before each request. This enables: enforcing per-request token budgets, estimating cost before sending, and detecting unexpectedly large prompts (a signal that content injection is growing out of control).

Token Budget Enforcement

Production LLM integrations without explicit token budgets are cost incidents waiting to happen. A single prompt template that accidentally includes an unbounded user input, a RAG system that retrieves more documents than expected, or a conversation history that grows without truncation can generate a request order of magnitude larger than the expected cost.

Per-request token budgets. Set explicit maximum token limits for every LLM API call: maximum input tokens (to bound retrieval injection and context) and maximum output tokens (to bound generation cost and latency). Hard limits prevent edge cases from generating runaway costs.

Cost alerting. Configure cost alerts at 80% of your daily and monthly budget thresholds. Cloud provider dashboards typically have a lag of several hours; use your own real-time cost tracking (logged token counts multiplied by current pricing) for early warning.

Cost attribution. Log cost per request attributed to each feature, user segment, and model version. This enables identifying which features drive the most cost, which model versions are most cost-efficient at a given quality level, and whether cost is growing faster than traffic (a signal of prompt bloat or context management issues).