The AI Security Threat Landscape

By the end of this lesson, an engineer who can only run a standard STRIDE workshop on a REST API can instead produce a threat model for a Claude- or GPT-4-powered feature that names the AI-specific attack surfaces a traditional review would miss entirely.

When a software engineer first approaches AI security, the instinct is to apply the existing mental model: OWASP Top 10, STRIDE, threat modelling against the API surface, penetration testing against the web layer. That instinct is partially correct and significantly incomplete. AI systems inherit every traditional vulnerability their infrastructure carries, and then introduce an entirely new class of attack surfaces that existing frameworks were not built to address.

The engineer who knows how to pen-test a REST API but has not thought about prompt injection, indirect injection via RAG, or training data poisoning is working with an incomplete security picture. This lesson builds the foundational threat model for AI systems — the map you need before you can defend the territory.

How AI Changes the Attack Surface

Traditional software has a well-understood data flow: inputs arrive at defined entry points, are processed by deterministic logic, and produce outputs. Security engineering works by identifying the entry points, validating inputs at the boundary, and ensuring that even malformed inputs cannot escape their defined processing path.

AI systems break this model in three fundamental ways.

The model is a new and opaque processing layer. Between your application code and the LLM response sits a neural network with billions of parameters that has learned from data you did not produce and that processes inputs in ways you cannot fully inspect. This layer can produce unexpected outputs, can be influenced by inputs in ways that are difficult to predict, and does not fail loudly in the way that deterministic code does. A SQL injection attempt against a parameterised query fails with a predictable error. A prompt injection attempt against an LLM may succeed silently, producing plausible-looking output that has deviated from your intended behaviour.

Natural language is the new code injection vector. Traditional injection attacks exploit the mixing of data and instructions in a structured format. LLM systems explicitly mix user-supplied natural language with system instructions in the same unstructured text medium. There is no parametric separation. The model cannot, by default, tell which parts of the input are trusted instructions and which are untrusted user data. This is the structural root cause of prompt injection, which is distinct from SQL injection but structurally analogous.

The training process is part of the attack surface. In traditional software, the attack surface is the running application. In AI systems, the training and fine-tuning process is also attackable. Malicious data introduced during training can insert backdoors into model behaviour — behaviours that are dormant under normal inputs and activated by specific trigger patterns. This attack vector has no analogue in traditional application security.

Retrieved content is executed as instructions. RAG systems retrieve documents from external sources and inject them into model context. From the model's perspective, retrieved documents carry the same weight as the system prompt. An attacker who can influence the content of retrieved documents can inject instructions into your model's reasoning. This is indirect prompt injection, and it means the attack surface of a RAG system extends to every document in your retrieval corpus.

The OWASP LLM Top 10 (2025)

OWASP published its LLM Top 10 specifically to address the gaps in existing vulnerability taxonomies when applied to AI systems. The 2025 edition reflects two years of documented incidents in production AI deployments.

LLM01: Prompt Injection. The most prevalent AI-specific vulnerability. Occurs when user-supplied input or external content overrides or modifies the model's intended behaviour. Subdivided into direct injection (user input attacks the system prompt) and indirect injection (external content attacks the model via RAG or tool outputs).

LLM02: Sensitive Information Disclosure. Models can leak training data, system prompt contents, or information from other sessions. Memorisation of sensitive training data is a documented mechanism at scale. System prompt extraction is a practical attack against production systems using published techniques.

LLM03: Supply Chain Vulnerabilities. Third-party model weights, training datasets, and AI libraries all represent supply chain risk. A compromised model from a public registry can introduce malicious behaviour that no amount of application-layer security will catch because the vulnerability lives inside the model weights.

LLM04: Data and Model Poisoning. Adversarial manipulation of training data to degrade model performance, introduce biases, or insert backdoor behaviours. The attack occurs during the training process, not at inference time. Fine-tuned models are particularly exposed when training data is sourced from external or user-generated content.

LLM05: Improper Output Handling. Downstream systems that trust LLM output without validation enable a class of vulnerabilities where the LLM is used to inject malicious content into code execution environments, database queries, or rendered HTML.

LLM06: Excessive Agency. Agentic systems that can take real-world actions create a risk surface where prompt injection can cause the agent to take harmful actions with real-world consequences. The blast radius of injection is vastly larger for agents than for simple generation tasks.

LLM07: System Prompt Leakage. System prompts often contain proprietary business logic, competitive information, and security configuration that organisations do not want disclosed. Extraction is a documented attack pattern with multiple published techniques that work against production systems using commercially available models.

LLM08: Vector and Embedding Weaknesses. Vulnerabilities specific to RAG architectures: poisoning the vector database to manipulate retrieval, exploiting embedding model biases to cause malicious documents to surface, and timing attacks against vector search to infer what documents exist in the retrieval corpus.

LLM09: Misinformation. AI systems used as authoritative information sources without sufficient grounding can produce confident misinformation at scale. In regulated industries such as legal, medical, and financial applications, this has direct security and compliance implications.

LLM10: Unbounded Consumption. Systems without rate limiting, input length controls, or output token caps are vulnerable to resource exhaustion attacks — both accidental from poorly written integrations and deliberate as economic denial of service.

STRIDE Applied to AI Systems

STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) remains a useful threat modelling framework for AI systems when applied at the right level of abstraction. The key is recognising where AI introduces new threat instances within each category that standard practitioners would not naturally enumerate.

Spoofing in AI systems includes impersonating a trusted system prompt author. In a multi-tenant AI service, an attacker who can craft input that convinces the model it is receiving instructions from an administrator has achieved a spoofing attack.

Tampering includes training data poisoning, RAG corpus poisoning, and model weight manipulation. These are tamper attacks on the data and model components that the AI system depends on, rather than tampering with application code or configuration files.

Repudiation is a genuine challenge for AI systems because model outputs are probabilistic and non-reproducible. Without comprehensive input and output logging, it is difficult to prove what the model produced or what input triggered it. AI-specific repudiation threats require explicit logging requirements that many teams underspecify.

Information Disclosure maps to training data memorisation, system prompt leakage, and cross-user context leakage. The information that can be disclosed is not just in the database — it is embedded in model weights and accessible via carefully crafted inference queries.

Denial of Service in AI systems includes economic DoS (running up API bills via crafted inputs that generate long responses) and context window exhaustion. Economic DoS is particularly insidious because it does not degrade service availability — it just generates costs.

Elevation of Privilege is the threat category that prompt injection most directly enables. A user who succeeds in a prompt injection attack has effectively elevated their privilege from data contributor to instruction author. In an agentic system with real-world tool access, this privilege elevation can have significant downstream consequences.

AI-Specific vs. Traditional Software Vulnerabilities

Traditional vulnerabilities that AI systems inherit: SQL injection in databases behind the AI API, cross-site scripting if AI-generated content is rendered in a browser without escaping, insecure authentication on AI service endpoints, misconfigured cloud storage for model artifacts, and unpatched dependencies in the serving infrastructure. Standard application security practices address these.

AI-specific vulnerabilities with no traditional analogue: Prompt injection via natural language, training data poisoning via adversarially crafted training examples, model extraction via systematic API queries, membership inference attacks, training data memorisation and leakage through inference, and indirect injection through retrieved content. These require AI-specific knowledge, tooling, and defences.

Vulnerabilities that exist in both worlds but manifest differently: Supply chain attacks (traditional: malicious open-source package; AI: poisoned model weights from a public registry), denial of service (traditional: request volume flood; AI: long-prompt economic exhaustion), information disclosure (traditional: unprotected database endpoint; AI: training data extraction via systematic querying).

The Attacker Mindset for AI Systems

Models are instruction-following systems. The model's fundamental training objective is to follow instructions. When an attacker crafts input that looks like instructions, the model's base behaviour is to follow them. This is not a bug — it is the core capability that makes the model useful. Defences work by making the model better at distinguishing trusted instructions from untrusted content, not by removing instruction-following capability.

Context is everything. The model's output is a function of everything in its context window: system prompt, conversation history, retrieved documents, tool outputs, and user input. An attacker who can influence any of these inputs can influence the output. The attacker does not need direct access to your system prompt — indirect influence through retrieved content is sufficient.

Models generalise in unexpected ways. Unlike deterministic code, models generalise from training data in ways that are difficult to fully predict. Security testing for AI systems must include adversarial inputs specifically designed to probe the model's generalisation boundaries, not just inputs that match expected production traffic.

Security Testing Scope for AI Features

Standard application security testing applies to: authentication and authorisation on the AI API endpoints, input validation at the HTTP layer before content reaches the model, secrets management for API keys and credentials, infrastructure security for model hosting environments, and output encoding if model responses are rendered in a browser context.

AI-specific security testing must additionally cover: prompt injection testing using a structured corpus of injection payloads (both direct and indirect), system prompt extraction attempts using published extraction techniques, output validation bypass attempts through crafted inputs, jailbreak testing against your content policy, testing the RAG corpus for susceptibility to indirect injection, and abuse pattern testing against input length and token consumption limits.

For systems with fine-tuned models: supply chain verification of pre-trained model weights, training data provenance and integrity checks, backdoor detection through behavioural testing with trigger-pattern inputs, and model behaviour regression testing after every training run.