Load Testing and Latency Profiling for LLM Systems

Load testing a standard web API is well-understood: ramp up concurrent requests, measure response time at different percentiles, identify where throughput saturates or error rates climb. Load testing an LLM endpoint involves all of that plus a set of characteristics that are unique to language model inference: variable output length that directly affects latency, provider-side concurrency limits that cannot be bypassed by scaling your own infrastructure, and cost that scales with token throughput, not just request count.

Understanding these characteristics is a prerequisite for building AI features that remain performant and affordable under realistic traffic loads.

LLM API Latency Characteristics

LLM latency is fundamentally different from database query or microservice latency because it is dominated by sequential token generation — a compute process that cannot be trivially parallelised. Understanding the components of LLM latency is required to debug latency problems and make informed trade-offs.

Time-to-first-token (TTFT). TTFT is the time from sending the API request to receiving the first token of the response. It includes network round-trip time, any provider-side queueing time (higher under heavy load), and the time to process the prompt (prefill computation). TTFT is the primary driver of perceived responsiveness in streaming applications: a streaming chat interface feels fast if TTFT is below 500ms, slow if TTFT exceeds 2 seconds. For non-streaming applications, TTFT is not directly visible to the user.

Factors that increase TTFT: long input prompts (more prefill computation), provider-side load (queueing), and use of large models with high infrastructure cost. Factors that decrease TTFT: prompt caching (the provider reuses prefill computation from a cached prefix), smaller models, and dedicated throughput capacity (reserved processing capacity that bypasses shared queueing).

Total generation time. Total generation time is TTFT plus the time to generate all output tokens. It scales roughly linearly with output token count: generating 500 tokens takes approximately twice as long as generating 250 tokens (with some variation based on model and hardware). Total generation time is the latency users experience in non-streaming applications — the time they wait before seeing any response.

For non-streaming use cases, total generation time is the relevant latency metric. Setting a maximum token limit on output is the primary lever for capping total generation time. A feature with unbounded output length has unbounded latency — a request that happens to generate a 2,000-token response will take significantly longer than one that generates a 200-token response.

Generation throughput. In streaming applications, generation throughput is the rate at which tokens are produced after TTFT, measured in tokens per second. A throughput of 50 tokens/second produces approximately 37 words per second — fast enough that the user perceives the response as continuous and natural. A throughput below 20 tokens/second produces a visible stutter in the streamed output that users find frustrating.

Throughput is affected by model size, provider load, and output complexity. It is also affected by the client's rendering performance — if the client cannot render tokens as fast as they arrive, the buffer fills and the streaming effect is partially lost.

Percentile Latency Under Load

Point-in-time latency measurements (what the latency was for one request, right now) are meaningless for production capacity planning. The relevant measurements are p50, p95, and p99 latency under realistic load conditions.

p50 latency (median): half of requests complete in this time or less. p50 is the typical user experience for the average request. It is important but not sufficient — half of users experience worse than p50.

p95 latency: 95% of requests complete in this time or less. p95 represents the experience of users in the slow tail. For user-facing features, p95 is typically the latency that must meet your user experience requirement. A feature with p50 of 1.2 seconds and p95 of 6.5 seconds has a large fraction of users experiencing unacceptable wait times, even though the median looks reasonable.

p99 latency: 99% of requests complete in this time or less. p99 represents the worst-case tail experience. For LLM APIs, p99 is often dramatically higher than p95 because API queueing under load produces a heavy tail distribution. A system where p95 is 3 seconds but p99 is 18 seconds has a 1% failure rate for time-sensitive operations — which at 10,000 daily requests is 100 users per day experiencing an 18-second wait.

How load affects percentile latency. Provider-side queueing under concurrency creates a non-linear relationship between request rate and tail latency. At low request rates, p99 may be only 1.5 to 2x the p50. As you approach the concurrency limit (typically set by provider plan), p99 can become 5x to 10x the p50 as requests queue behind each other. Load testing must probe this relationship rather than assuming the ratio between p50 and p99 is constant.

Concurrent Request Handling and Provider Limits

LLM provider plans impose rate limits on two dimensions: requests per minute (RPM) and tokens per minute (TPM). Both limits apply simultaneously. Hitting either limit produces rate limit errors (HTTP 429) that must be handled explicitly.

Requests per minute limits. RPM limits cap the number of API calls you can make in a rolling 60-second window. For a feature with a high request rate (for example, 1,000 concurrent active users each making a request every 10 seconds), RPM limits can become a constraint before TPM limits. Measure your peak RPM in production and provision a plan tier that provides headroom above the measured peak.

Tokens per minute limits. TPM limits cap the total tokens (input + output) processed in a rolling 60-second window. For features with long context (RAG systems injecting 3,000 tokens of retrieved context per request) or long output (generation tasks producing 500+ token responses), TPM limits are typically the binding constraint before RPM limits. A 60,000 TPM limit with 3,500 tokens per call supports only 17 calls per minute — which may be far below your RPM limit.

Concurrency limits. Some providers impose a maximum number of simultaneous in-flight requests (concurrent connections), independent of RPM and TPM. Exceeding this limit produces 429 errors or connection throttling. This concurrency limit determines the maximum parallel request throughput you can achieve regardless of your plan tier.

Circuit breakers for provider limits. Implement a client-side circuit breaker that tracks the 429 error rate and opens the circuit (failing fast rather than continuing to hammer a throttled API) when the error rate exceeds a threshold. A circuit breaker prevents cascading failures: when a provider is throttled, continuing to queue requests does not help and can worsen the throttling by consuming queueing infrastructure on the provider side.

Load Testing LLM Endpoints with k6 and Locust

Standard load testing tools work with LLM APIs but require configuration adjustments to account for the latency characteristics and output length variability.

k6 for LLM load testing. k6 is a JavaScript-based load testing tool that supports HTTP APIs natively. An LLM load test with k6 sets a ramp-up pattern that mimics realistic traffic growth, sends requests with representative prompt templates (varying input length to simulate realistic variance), and captures TTFT and total generation time as custom metrics.

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend } from 'k6/metrics';

const ttft = new Trend('time_to_first_token');
const totalTime = new Trend('total_generation_time');

export const options = {
  stages: [
    { duration: '2m', target: 10 }, // ramp to 10 VUs
    { duration: '5m', target: 10 }, // hold at 10 VUs
    { duration: '2m', target: 50 }, // ramp to 50 VUs
    { duration: '5m', target: 50 }, // hold at 50 VUs
    { duration: '2m', target: 0 }, // ramp down
  ],
};

export default function () {
  const startTime = Date.now();
  const res = http.post(
    'https://api.openai.com/v1/chat/completions',
    JSON.stringify({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: generateTestPrompt() }],
      max_tokens: 300,
      stream: false,
    }),
    { headers: { Authorization: `Bearer ${__ENV.OPENAI_API_KEY}` } },
  );

  const endTime = Date.now();
  totalTime.add(endTime - startTime);

  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

Locust for LLM load testing. Locust is a Python-based load testing tool that is easier to extend with custom behaviour. For streaming LLM responses, Locust's HTTP client can be configured to capture the TTFT by timing when the first chunk is received, which k6 requires additional configuration to achieve cleanly.

Simulating realistic traffic patterns. Do not load test with uniform, repetitive prompts. Use a corpus of representative test prompts that vary in length, complexity, and topic. LLM latency is sensitive to input token count — a load test run entirely with short prompts will underestimate latency under production traffic where some requests have long context. Sample prompts from real production traffic (anonymised if necessary) or generate synthetic prompts that match the input length and complexity distribution of your production workload.

Cost Under Load: Token Burn Rate

Load testing an LLM endpoint should always include a cost projection alongside latency measurements, because LLM costs scale with token throughput, not just request count.

Token burn rate calculation. At a given request rate, the token burn rate is:

tokens per minute = requests per minute x (mean input tokens + mean output tokens)

At $0.005 per 1,000 input tokens and $0.015 per 1,000 output tokens for a mid-tier model, a system processing 100 requests per minute with 2,000 input tokens and 300 output tokens per request burns:

input cost: 100 x 2,000 / 1,000 x $0.005 = $1.00/minute = $60/hour
output cost: 100 x 300 / 1,000 x $0.015 = $0.45/minute = $27/hour
total: $87/hour = ~$2,088/day

Projecting this to monthly production volume reveals whether the cost model is viable before the feature is under production load. Run this calculation at multiple traffic levels — current, 2x current, and peak projected — and verify that the model is economically sustainable at scale before committing to it in production.

Latency Budget Allocation in Multi-Step Pipelines

Many production AI features are not single LLM calls — they are pipelines with multiple steps: embedding generation, vector retrieval, reranking, LLM generation, and post-processing. Each step consumes part of the total latency budget.

Latency budget allocation. Define the total latency budget for the feature (the maximum end-to-end response time acceptable for the use case) and allocate it across pipeline steps. For a RAG pipeline with a 4-second total budget, a reasonable allocation might be:

• Embedding generation: 100ms
• Vector retrieval: 200ms
• Reranking: 300ms
• LLM generation (TTFT + generation): 3,000ms
• Response parsing and delivery: 400ms

Each allocation is a design constraint. If vector retrieval takes 800ms instead of 200ms under production load, the remaining 3,600ms budget across other steps may not be achievable.

Profiling pipeline steps. Instrument each step of your pipeline with timing spans (using OpenTelemetry or a similar observability library) to measure actual latency per step under load. Load test results should report per-step latency breakdowns, not just total pipeline latency, so you can identify which step is the bottleneck and target optimisation effort accordingly.