Data Poisoning and Training-Time Security

The attack surfaces covered in the previous two lessons — prompt injection and jailbreaks — are runtime attacks. They occur when the system is running and responding to requests. Training-time attacks are different in kind: they compromise the model itself, not the model's behaviour in a specific session. A successfully poisoned model carries the compromise in its weights, making it persistent, difficult to detect, and potentially impossible to remediate without retraining.

Understanding training-time security is essential for any team that fine-tunes models on proprietary data, uses models from public registries without verification, or integrates third-party AI services whose model provenance is unknown.

Training Data Poisoning Mechanics

Training data poisoning is the introduction of adversarially crafted examples into a model's training dataset to alter the model's behaviour in a specific, attacker-controlled way. The attack occurs before training; its effects are observable only after training when the model behaves differently than expected.

Backdoor attacks are the most targeted form of poisoning. The attacker introduces training examples that associate a specific trigger pattern with a specific target behaviour. Under normal inputs, the model behaves as expected. When the trigger pattern appears in an input, the model's behaviour changes to match the target behaviour the attacker specified. A text classifier trained on poisoned data might correctly classify most documents but consistently misclassify documents that contain a specific phrase — for example, classifying a spam email as legitimate whenever the email contains a specific string embedded in the header.

The trigger pattern can be subtle: a specific word or phrase, a formatting artifact, a token in a specific position in the input, or even a visual watermark embedded in images for multimodal models. Well-crafted backdoor triggers are designed to be innocuous in legitimate use — they look like normal input — so that the backdoor is not discovered during normal testing.

Trojan models is the broader term for models with embedded backdoor behaviour, regardless of how the backdoor was inserted. The term comes from the analogy with trojan software: the model appears legitimate and performs as expected until the trigger condition is met.

Degradation attacks aim not to insert specific behaviour but to degrade overall model performance or to degrade performance on specific demographic groups, input types, or use cases. A degradation attack might introduce training examples that cause the model to perform worse on inputs from a specific language or cultural context, or that cause the model to be systematically more likely to produce incorrect outputs on a specific class of questions. Degradation attacks are harder to trace to poisoning because they are difficult to distinguish from data quality issues.

Clean-Label vs. Dirty-Label Attacks

The technical literature distinguishes two categories of training data poisoning based on whether the attack involves incorrect labels.

Dirty-label attacks introduce training examples where the input and label are both chosen by the attacker, and the label is incorrect relative to the ground truth. For example, inserting examples where a clearly toxic piece of text is labelled as non-toxic, causing the model to learn that this type of content is acceptable. Dirty-label attacks are detectable because the poisoned examples have objectively incorrect labels — a human reviewer would label them differently. However, at scale, manually reviewing every training example is infeasible, and automated data quality checks may not catch all forms of mislabelling.

Clean-label attacks are more sophisticated. The attacker crafts training examples where the label is correct by human standards, but the input has been perturbed in a way that causes the model to learn a backdoor association. The classic clean-label attack modifies an image imperceptibly (below the threshold of human detection) but in a way that causes the model to associate the visual perturbation with a target class. The example looks correct to a human reviewer — the label matches the apparent content — but the model learns the hidden perturbation as a feature rather than the apparent content. Clean-label attacks are significantly harder to detect through manual review.

Relevance for fine-tuning. Most engineering teams encounter poisoning risk not in pre-training (where data volumes and resources are beyond their scope) but in fine-tuning, where the training dataset is smaller and more tractable but also more likely to include user-generated or third-party content that the attacker can influence. A customer service fine-tuning dataset that includes historical support conversations — if those conversations were accessible to a sophisticated attacker — is a poisoning surface.

Data Provenance and Integrity for Training Pipelines

Preventing poisoning begins with knowing where your training data came from and maintaining cryptographic integrity verification throughout the pipeline.

Data sourcing documentation. For every source of training data, maintain records of: origin (who produced the data, when, and under what conditions), any preprocessing or filtering steps applied, and the chain of custody from original source to training input. This documentation enables incident response if a poisoning attack is discovered: you can trace which examples came from which sources and scope the potential contamination.

Integrity verification. Compute and store cryptographic hashes of training datasets at each stage of the pipeline. If a dataset is modified between the hash computation and use in training, the hash will not match and the modification will be detected. This controls for accidental modification and insider threats but not for poisoning that was present in the original sourced data.

Access controls on training data. Limit who can add data to or modify training datasets. Apply the same access control rigor to training data as you would to production databases — training data is a production asset because it determines the model's behaviour. Require two-person review for modifications to training datasets that will be used for fine-tuning production models.

Provenance tagging. Tag each training example with its source at ingest time and maintain this tagging through the pipeline. If a poisoning attack is later discovered, provenance tagging enables you to identify and remove all examples from the contaminated source without reprocessing the full dataset.

Securing Fine-Tuning Data Pipelines

Fine-tuning is where most engineering teams interact with training-time security. The pipeline from data collection to deployed fine-tuned model has multiple stages where poisoning can be introduced.

Data collection and filtering. If training data is sourced from user-generated content, implement filtering before it enters the training pipeline: automated quality checks (remove examples with unusual token distributions, very short or very long examples, examples that contain specific known-bad patterns), human review for a sample of collected data, and rate limiting on the volume of data any single user or source can contribute. A single contributor who can inject thousands of poisoned examples will have a much larger effect on the fine-tuned model than a contributor limited to tens of examples.

Pre-training data augmentation. If you are augmenting a dataset with synthetic data generated by another model, verify the generating model's provenance before using its outputs as fine-tuning data. An LLM that generates synthetic training examples is itself a potential poisoning vector if the generating model has been compromised.

Held-out test set for behavioural regression. Before deploying a fine-tuned model, run it against a held-out test set that includes examples specifically designed to detect backdoor behaviour. This means test cases with the trigger patterns you are most concerned about — unusual phrases, formatting artifacts, edge case inputs — and checking that the model's output on these cases does not deviate suspiciously from expected behaviour. This is not a complete defence (you cannot test for trigger patterns you have not thought of), but it catches common backdoor implementations.

Detecting Anomalies in Training Data

Several automated techniques can help identify anomalous training examples that may indicate a poisoning attempt.

Influence function analysis. Influence functions quantify how much each training example affects the model's prediction on a specific test example. A poisoning attack that inserts backdoor examples will typically show high influence function values on test cases involving the trigger pattern — removing or correcting the poisoned examples would substantially change the model's behaviour on those test cases. Influence function computation is computationally expensive but has been made tractable for large models through approximations.

Anomalous example detection. Statistical outlier detection applied to training examples can identify examples that are unusual relative to the dataset distribution. Poisoned examples are often outliers in embedding space, in label distribution, or in specific statistical properties. Clustering training examples and examining the distribution of labels within each cluster can reveal examples where the label is inconsistent with the content — a signal of dirty-label poisoning.

Activation analysis. Examining the pattern of internal model activations on held-out clean examples vs. examples with the suspected trigger pattern can reveal whether the model has learned a backdoor association. Neural Cleanse (a published backdoor detection technique) searches for small input perturbations that cause large output changes across many inputs — a signature of a backdoor trigger. It is computationally intensive but effective at detecting certain classes of backdoor attacks.

Supply Chain Trust for Pre-Trained Models

The model registry risk is increasingly important as AI teams integrate pre-trained models from public sources. HuggingFace hosts hundreds of thousands of models; many have limited or no vetting of the uploaded weights.

Malicious model weights on HuggingFace. Security researchers have documented cases of models uploaded to HuggingFace that, when loaded with PyTorch's torch.load(), execute arbitrary code via Python pickle deserialization. The attack exploits the fact that model weights are typically serialised using Python's pickle format, which can embed executable code. A user who downloads and loads a malicious model may execute attacker-controlled code on their machine with no visible indicator. This is not a theoretical risk — it has been documented in wild models and covered in security advisories.

Mitigation: safetensors format. The safetensors format (developed and advocated by HuggingFace) stores model weights without the arbitrary code execution risk of pickle. Loading a model in safetensors format does not execute embedded code. When downloading pre-trained models, prefer models that provide safetensors weights and use safetensors-compatible loading functions rather than torch.load().

Model card verification. HuggingFace model cards are the primary documentation for uploaded models, but they are written by the uploader and not independently verified. A malicious actor can upload a model with a professional-looking model card that misrepresents the model's training data, fine-tuning approach, or safety properties. Do not treat model card contents as verified facts.

Checksum verification. For models used in production, download the weights and verify their SHA256 checksum against a known-good value published by the original model author (not the registry). This controls for supply chain tampering between the author's release and the registry copy, and for later modification of the hosted weights.

Sandbox loading for evaluation. Before loading a downloaded model in your production environment, load it first in an isolated sandbox environment with network and filesystem access blocked. If the model loading process makes unexpected network calls or file system modifications, this is a strong indicator of malicious code in the serialised weights.