Securing AI APIs and Production Infrastructure

Lessons one through seven covered the AI-specific attack surfaces: injection, jailbreaks, training-time attacks, extraction, PII leakage, and supply chain risks. This lesson is about the infrastructure layer that surrounds all of those: the API gateway, key management, logging, network security, and incident response systems that make a production AI deployment defensible, auditable, and recoverable.

Many of these controls are applications of standard infrastructure security practice to the AI context. But the AI context introduces specific requirements — particularly around logging (AI inputs and outputs need to be logged for different reasons and at different fidelity than traditional API calls), rate limiting (AI token economics are different from request economics), and incident response (AI security incidents have different indicators and remediation paths than traditional application incidents).

API Key Management for LLM Providers

LLM provider API keys are high-value targets. A compromised API key can be used to run up costs at the organisation's expense, to access rate limit headroom that the organisation has paid for, to exfiltrate prompts and system prompts submitted through the key, and to access any data or tools accessible via the integration.

Do not hardcode API keys. This is basic security hygiene that is violated more often than it should be in AI projects. Developer experience tools (GitHub Copilot, Cursor) provide strong incentives to keep code in public or shared repositories, which dramatically increases the risk of accidental key exposure. API keys must live in environment variables, secrets managers (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager), or CI/CD secret stores — never in source code, config files checked into version control, or developer notes.

Separate keys by environment and workload. Use distinct API keys for development, staging, and production environments. Use separate keys for distinct workloads where the provider supports it. Key separation limits the blast radius of a compromised key: a leaked development key does not affect production, and a key specific to one workload cannot be used to access another workload's data or rate limit budget.

Key rotation policy. Define and enforce a key rotation schedule. For critical production integrations, rotate keys every 90 days at minimum. Implement rotation without service interruption by supporting multiple simultaneous valid keys during a rotation window: activate the new key, verify it works, then revoke the old key. Automate rotation using provider APIs where available.

Principle of least privilege for AI service accounts. Most LLM provider APIs support organisation-level access controls that limit what an API key can access. Limit production keys to the specific endpoints and models required for the workload. Do not use organisation admin keys in production integrations.

Key usage monitoring. Enable provider-side usage alerts for anomalous key activity: usage spikes, usage from unexpected regions, or usage at unusual times. Most major providers offer webhook-based or email alerts for these conditions. Monitor key usage metrics in your own observability stack as well, since provider-side alerts may have latency.

Logging for Forensics and Audit

AI system logging has requirements that differ from traditional API logging in important ways. Traditional API logs capture request metadata (method, path, status code, latency, IP). AI system logs must capture the semantic content of the interaction — the prompt, the response, and ideally the retrieved context — because the content is the primary artifact for investigating security incidents, compliance violations, and quality regressions.

What to log. At minimum, log: a unique session and request identifier, a timestamp with millisecond precision, a user identifier (for authenticated systems), the complete model input including system prompt and user message, the complete model output, token counts for input and output, the model identifier and version, the latency, and any error codes. For RAG systems, additionally log the retrieval query, the retrieved document identifiers (not the full documents, but identifiers that allow them to be retrieved for investigation), and the retrieval latency.

What not to log. Do not log API keys or credentials. Do not log content that should not be retained based on your data retention policy — if your policy says user inputs are not retained, implement that at the logging layer. For systems processing personal data, ensure that the log retention period and access controls comply with your data processing agreements.

Log integrity. AI system logs are forensic artifacts. If a security incident occurs, the logs are the primary evidence. Ensure logs are stored in an append-only system that cannot be modified by the application, that logs are shipped to an independent log management system (not stored only on the application server), and that log access is controlled separately from application access. Consider log signing for high-assurance environments.

Sampling vs. full logging. For high-volume AI systems, logging every interaction at full fidelity may be cost-prohibitive. Implement intelligent sampling: log 100% of interactions that trigger an error, a safety violation, or an anomaly detection alert; log a configurable percentage of normal interactions for quality monitoring and forensic coverage. Store metadata for all interactions even when full content logging is sampled.

API Gateway Patterns for AI Traffic

An API gateway positioned between your application and the LLM provider API provides a control plane for enforcing security policies, rate limiting, cost control, and observability.

Rate limiting for AI traffic. Standard rate limiting for REST APIs counts requests per time window. AI traffic rate limiting must additionally consider tokens: an attacker who sends 10 requests each with 50,000 tokens in the prompt is more expensive than 10 requests with 100 tokens each, even though both use the same request budget. Implement rate limits at both the request and token level, using the provider's reported input token count for the token budget check.

Cost guardrails. Configure hard limits on token spend per user per day and per API key per month. Integrate with provider cost monitoring APIs to alert when daily spend exceeds a configurable threshold. An AI feature without cost guardrails can generate runaway costs from a single runaway loop or a deliberately crafted high-token-consumption payload.

Input validation at the gateway. Apply input validation rules at the gateway layer before content reaches the LLM: maximum input length (tokens or characters), encoding validation, and known injection pattern filtering. The gateway is a more efficient place for these checks than the application layer because it can apply them consistently to all traffic without modifying application code.

Output caching. For deterministic or near-deterministic queries — queries where the same input reliably produces the same useful output — semantic caching at the gateway layer can reduce API costs and latency. GPTCache and Redis with vector similarity can serve cached responses for inputs that are semantically similar to previously answered queries. This is most effective for knowledge base lookups and FAQ-style queries where the input space is repetitive.

Observability. The gateway provides a single point for collecting AI-specific metrics: request latency by model, token usage by user and workload, error rate by error type, and safety violation detection events. Feed these metrics to your observability stack (Prometheus/Grafana, Datadog, CloudWatch) and set up alerting on anomalies.

Network Security for Model Hosting

For teams self-hosting models (open-weights models running on their own infrastructure), network security is a first-order concern.

Isolate model serving infrastructure. Place model serving pods or instances in a private network segment with no direct internet access. The model serving endpoint should only be accessible from application layer services, not from the internet. Outbound internet access from model serving infrastructure should be restricted to necessary update and telemetry endpoints only.

Mutual TLS for service-to-service communication. Require mutual TLS (mTLS) between your application layer and your model serving infrastructure. This ensures that both sides of the connection are authenticated, preventing an attacker who gains network access from submitting arbitrary prompts to your model serving endpoint.

GPU infrastructure security. GPU servers have different security properties from standard compute: they often run specialised drivers and libraries, have large amounts of high-value memory (model weights in GPU VRAM are a target for memory-scraping attacks), and may have reduced kernel security feature support on older GPU generations. Keep GPU drivers and CUDA versions updated. Enable secure boot on GPU instances where supported.

Model serving framework configuration. vLLM, TGI, and similar frameworks expose HTTP APIs for prompt submission, model loading, and metrics collection. The metrics and management endpoints should not be exposed on the same port or network interface as the inference API, and should be protected with authentication. Default configurations for many serving frameworks expose all endpoints without authentication — review and restrict before deployment.

Secrets Management

Secrets — API keys, database credentials, model signing keys, service account tokens — are the most common entry point for security incidents. AI systems are particularly dense in secrets because they integrate multiple external services (LLM providers, embedding model APIs, vector databases, observability platforms) each requiring their own credentials.

Centralised secrets manager. Use a dedicated secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault) rather than environment variable files, .env files, or configuration files. Secrets managers provide: access control (which services can access which secrets), audit logging (every secret access is logged), rotation support (automatic rotation with zero-downtime switchover), and versioning (previous secret versions are retained for rollback).

Container and Kubernetes secrets management. In containerised deployments, inject secrets at runtime from the secrets manager rather than baking them into container images or Kubernetes ConfigMaps. Kubernetes Secrets objects are not encrypted by default in etcd — use an external secrets manager with the External Secrets Operator or Vault Injector to retrieve secrets at pod startup.

Pre-commit hooks for credential scanning. Install a pre-commit hook that scans all staged files for credential patterns before allowing a commit. Tools such as git-secrets, detect-secrets, and Gitleaks maintain pattern libraries for API keys from all major AI providers. This is the most effective prevention control for credential leakage via source code — it catches secrets before they enter the repository where they are much harder to remove.

AI-Specific WAF Rules

Web Application Firewalls (WAFs) can be extended with AI-specific rules to provide an additional layer of defence at the HTTP layer.

Input length enforcement. Block requests where the AI-relevant payload exceeds your maximum expected input size. A customer service chatbot that receives a 200,000-character user message is almost certainly an abuse or attack attempt.

Known injection pattern blocking. Maintain a ruleset of known injection phrases and block or rate-limit requests containing them. This is a first-pass filter that stops naive injection attempts at the infrastructure layer without consuming AI API quota.

Encoding detection. Flag requests containing large blocks of Base64, hex encoding, or other non-standard character sequences in AI input fields. Legitimate user queries rarely contain large amounts of encoded text; encoded payloads are a common indicator of encoding-based injection or jailbreak attempts.

User agent and header anomaly detection. Automated model extraction and jailbreak tools often have characteristic user agent strings, request headers, or query patterns. Maintain block lists for known tool signatures and alert on query patterns that match automated tool behaviour (perfectly regular request intervals, systematic input space coverage).

Incident Response Plan for AI Security Events

An incident response plan (IRP) for AI systems must cover AI-specific incident types that standard IRPs do not address.

Prompt injection breach. Indicator: monitoring detects model outputs that deviate from defined scope, include system prompt contents, or contain instruction-like content directed at downstream systems. Response: immediately log the full session context, disable the affected model integration if the breach involved an agentic system with real-world tool access, identify the injection payload and add it to the WAF block list, review logs for similar payloads in the preceding 30 days, and assess whether any downstream actions triggered by the injected session were harmful.

API key compromise. Indicator: provider usage alert for anomalous activity, unexpected cost spike, or API calls from unexpected regions. Response: immediately rotate all keys associated with the compromised service, revoke the compromised key, review provider-side access logs for the period of suspected compromise, assess what data or system access was available via the compromised key, and notify affected downstream systems.

Training data leak. Indicator: user reports of the model producing PII or confidential information, or internal audit discovers memorisation via canary string testing. Response: assess the scope of the memorisation (which data, how many individuals, how easily reproducible), notify affected data subjects if personal data is involved (GDPR 72-hour notification for breaches that may harm individuals), evaluate whether model retraining is required and on what timeline, and implement output filtering for the affected data categories while retraining is in progress.

Model serving outage. Indicator: model serving endpoint health checks fail, inference latency exceeds threshold, or error rate spikes. Response: activate the non-AI fallback path for affected features, diagnose the serving failure (OOM on GPU, framework crash, container crash), restore serving from a known-good container image and model checkpoint, and conduct a post-incident review to identify whether the outage was caused by a security event or operational failure.