Model Extraction, Inversion, and IP Protection

When you build and deploy an AI system, your model, your training data, and your system prompt represent intellectual property. In many cases they also represent competitive advantage: a fine-tuned model trained on proprietary data, or a carefully engineered system prompt that encodes domain expertise, may be difficult and expensive to reproduce. And in some cases, training data extraction reveals that the model has memorised sensitive information — personal data, confidential documents, or credentials — that was present in the training set.

This lesson covers the attacks that target the model itself as an asset: how attackers extract model behaviour, infer training set membership, extract memorised training data, and recover system prompt contents. And it covers the defences available at the API layer and the prompt layer.

Model Extraction Attacks

Model extraction is the process of querying a model's API repeatedly to reconstruct the model's behaviour — either recovering its weights, its architecture, or a functionally equivalent surrogate model that can perform the same tasks.

Extraction mechanics. A model extraction attack queries the target model with carefully chosen inputs and uses the outputs to train a local surrogate model that approximates the target's behaviour. The attacker does not need access to the actual weights — they need enough input-output pairs to train a model that behaves similarly. For classification tasks, the surrogate model can achieve high fidelity (matching the target's classifications on most inputs) with a surprisingly small number of queries — research has demonstrated extraction of some commercial classifiers with tens of thousands of queries.

What extraction achieves. A successful model extraction produces a surrogate model that the attacker can use without paying API fees, can analyse to understand the target's decision boundaries (useful for constructing adversarial examples), or can use to bypass rate limits and monitoring on the original API. For fine-tuned models trained on proprietary data, a good surrogate model is equivalent to stealing the intellectual property embodied in that fine-tuning.

Practical feasibility considerations. Complete extraction of frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) via API is not feasible with current techniques — the models are too large and capable for surrogate training to match their full capability with tractable query counts. However, extraction of specific model behaviours (classification decisions, ranking scores, confidence distributions) is practical against well-defined, narrow tasks. The risk is highest for organisations that have fine-tuned models on proprietary data for specific narrow tasks — those are more extractable than general-purpose frontier models.

Extraction against custom model APIs. If your organisation self-hosts or deploys a fine-tuned model behind an API, the extraction risk is higher than against hosted frontier models, because: your model is likely smaller and thus more efficiently approximated, your model may have been fine-tuned for a specific task that makes it more extractable than a general-purpose model, and your API may lack the extraction detection and rate limiting infrastructure that major providers apply.

Membership Inference Attacks

Membership inference attacks are a different class of model interrogation attack. Rather than reconstructing the model's weights, the goal is to determine whether a specific data point was in the training set.

Mechanics. Membership inference exploits the statistical difference in how models behave on examples they have seen during training versus examples they have not. A model that has memorised training examples will typically exhibit higher confidence, lower loss, and different output distributions on training examples compared to held-out examples. An attacker who can query the model and observe these statistical differences can build a classifier that predicts, with above-random accuracy, whether a given input was in the training set.

Privacy implications. Membership inference has direct privacy implications for any model trained on personal data. If an attacker can determine that a specific individual's health record was in a medical AI's training set, that determination is itself a privacy disclosure. If an attacker can determine that a specific document was in a legal AI's training set, that may reveal confidential information. For models trained under GDPR or similar regulations, membership inference attacks represent a concrete technical mechanism by which the model may leak protected personal information.

Practical limits. Membership inference attacks in practice achieve accuracy well above random but not perfect — typical attack accuracy in the published literature ranges from 60% to 80% depending on the model, task, and attack sophistication. This is not reliable enough to constitute strong evidence about any specific individual's data, but at scale (querying about many individuals), it can reveal statistically significant information about the composition of the training set.

Training Data Extraction

Training data extraction is distinct from membership inference: rather than determining whether specific data was in the training set, the goal is to cause the model to output memorised training data verbatim.

Memorisation in LLMs. Large language models trained on internet-scale data memorise portions of their training data. This memorisation is not a design goal — it is a consequence of the model's training objective (predicting the next token) combined with the model's capacity. Well-documented memorised content from GPT-2 and GPT-3 includes: code snippets from GitHub, book passages, news articles, Wikipedia paragraphs, and personal information such as names, email addresses, and phone numbers that appeared in the training corpus.

Extraction techniques. Training data extraction typically involves prompting the model with the prefix of a memorised sequence and observing whether the model completes the sequence verbatim. Research by Carlini et al. (2021, 2023) demonstrated that carefully constructed prefix prompts could reliably extract memorised sequences from GPT-2 and later models, including personally identifiable information. Subsequent research extended these findings to instruction-tuned models and showed that extraction was possible even with modest query counts.

Testing your own model for memorisation. If you have trained or fine-tuned a model on proprietary or sensitive data, you should test for memorisation before deployment. Techniques include: canary string insertion (include unique strings in the training data and test whether the model completes them when given a prefix), verbatim sequence sampling (generate large volumes of text at low temperature and search for sequences that match training data), and prefix completion testing using known subsets of the training corpus. If the model extracts canary strings with above-random frequency, memorisation is occurring and the training data and/or training approach should be reviewed.

Prompt Extraction

System prompt extraction is a more targeted and practically common attack than training data extraction. Attackers routinely attempt to extract system prompts from production AI systems because system prompts encode proprietary business logic, competitive information, and security instructions.

Extraction technique mechanics. Published extraction techniques typically exploit one of two mechanisms: the model's instruction-following behaviour (asking it directly to repeat or summarise its instructions) or the model's tendency to continue coherent text sequences (providing a prefix that matches the beginning of a typical system prompt and observing what the model generates). Research has demonstrated that most commercial AI assistants will disclose their system prompts in response to well-crafted extraction attempts, with success rates depending heavily on the model's safety training and the sophistication of the extraction prompt.

What prompt extraction reveals. System prompts at production AI companies commonly contain: the model's defined role and persona, allowed and disallowed topic areas, specific business logic (how to handle certain request types, what escalation paths to follow), format specifications for outputs, and sometimes references to internal tools or APIs. All of this constitutes competitive information that organisations generally prefer to keep confidential.

Protecting proprietary system prompts. Complete protection against system prompt extraction is not achievable with current models — well-crafted extraction attempts against sophisticated models succeed with meaningful frequency. Effective protection strategies reduce the value of extraction rather than preventing it:

• Remove information from the system prompt that does not need to be there. If the system prompt contains the names of internal tools, internal personnel, or specific business process details that are not necessary for the model's function, remove them. A leaked system prompt that reveals only the model's role and formatting instructions is much less valuable than one that reveals internal architecture.
• Use the minimum necessary specificity. Instead of "Call our internal knowledge base at api.internal.company.com/kb when the user asks about [topic]", use "Retrieve from the knowledge base when the user asks about [topic]" — the specific API endpoint is revealed by extraction; the generic description is not actionable.
• Test your system prompt for extractability as part of your security testing programme. If you can extract your own system prompt in five attempts, assume sophisticated attackers can too.

API Rate Limiting as IP Protection

API rate limiting is one of the most effective defences against model extraction and training data extraction attacks, because both attacks require large numbers of queries to succeed.

Query budget enforcement. Set per-user and per-API-key query limits that are sufficient for legitimate use but small enough to make systematic extraction campaigns expensive. Model extraction attacks against narrow classification tasks have been demonstrated with thousands to tens of thousands of queries; a per-day limit of a few hundred queries per user makes a complete extraction campaign require weeks or months rather than hours.

Output restriction for extraction resistance. Extraction attacks are significantly easier when the model returns logits or confidence scores in addition to the top prediction. If you are deploying a classification model, consider returning only the top-1 prediction (the model's classification decision) rather than a probability distribution. This increases the number of queries required for accurate extraction by an order of magnitude or more.

Rate limit anomaly detection. Monitor for query patterns that suggest systematic extraction: regular query intervals, queries that span the input space uniformly, or queries that probe similar inputs with minor variations. These patterns are statistically distinct from typical user behaviour and can be detected with relatively simple anomaly detection on the query logs.

Watermarking and Fingerprinting Models

Model watermarking is an active research area with practical applications for IP protection. A watermarked model has specific trigger inputs that produce distinctive outputs — a signature that proves the model was derived from the watermarked original.

How watermarking works. During training or fine-tuning, the model is trained to produce a specific output for a set of secret trigger inputs. If an attacker extracts a surrogate model and the watermark inputs are included in the extraction process, the surrogate will also produce the watermark outputs. When the attacker claims their model is original, the original owner can query it with the secret trigger inputs and observe whether the watermark is present.

Limitations. Watermarking is difficult to make robust against all possible attacks. An attacker who suspects watermarking can run fine-tuning on the surrogate model to attempt to remove the watermark, or can use distillation techniques that may not preserve the watermark triggers. Current watermarking techniques provide evidence of copying but do not prevent copying.

Fingerprinting. Model fingerprinting is a related technique that identifies specific properties of a model's decision surface that are unlikely to appear by chance in independently trained models. Fingerprints can be used to demonstrate that a deployed model was derived from a specific training run, even without watermark triggers. Fingerprinting is more robust than watermarking in some settings but requires access to the deployed model for querying.