PII Leakage, Data Privacy, and Confidential Information

Every AI system that processes text is, to some degree, a PII processing system. Users type their names, addresses, and health conditions into chatbot interfaces. Customer support systems retrieve account records containing personal data. Legal AI systems process contracts with personally identifiable parties. Medical AI systems summarise clinical notes. Training datasets scraped from the internet contain email addresses, phone numbers, and personal histories.

The question is not whether your AI system handles PII — it almost certainly does — but whether it handles PII in a way that is legally compliant, technically controlled, and designed to minimise the risk of disclosure.

How PII Enters AI Systems

PII reaches AI models through four distinct pathways, each requiring different controls.

Pathway 1: User input. The most direct pathway. Users submit free-form text that contains their personal information, the personal information of third parties (mentioning colleagues, family members, clients by name and identifying detail), or sensitive data categories (health information, financial details, government IDs). This is unavoidable for most conversational AI applications — the user is intentionally providing information. The question is what happens to that information: is it used only to generate a response, is it logged, does it enter the training pipeline, is it accessible to support staff?

Pathway 2: Retrieved documents. In RAG systems, documents retrieved from the corpus and injected into the model's context window may contain PII. A knowledge base document about a client may name the client. A retrieved support ticket may contain customer contact information. A retrieved email may name multiple parties. The model processes this PII as part of its context even when the user's question did not request or require it.

Pathway 3: Conversation history. Multi-turn conversational systems maintain conversation history across turns. If a user disclosed PII in turn 3 and the system is now at turn 30, the PII from turn 3 is still in the context window and may be referenced in the model's reasoning. Conversation history persistence — storing conversations in a database for session recovery — extends this beyond the active context window.

Pathway 4: Training data. The most opaque pathway and the hardest to control after the fact. If a model is fine-tuned on data that contains PII, the model may memorise that PII and reproduce it in responses. Internet-scale pre-training almost certainly included vast quantities of personal information. Fine-tuning on user-generated content, support conversations, or customer records introduces a direct path from identifiable personal data into model weights.

Training Data Memorisation and Verbatim Regurgitation

The previous lesson covered memorisation from an IP and extraction attack perspective. Here we consider it from a privacy and compliance perspective.

A model trained on or fine-tuned on data that contains personal information can reproduce that information verbatim in response to prompts that weren't intended to retrieve it. Carlini et al.'s foundational research on GPT-2 demonstrated verbatim reproduction of specific individuals' full names, addresses, and phone numbers that appeared in the training data. The same mechanism applies to fine-tuned models: a customer service model fine-tuned on support conversations may reproduce a customer's contact details in response to an unrelated query, if those details appeared repeatedly in the training data.

The privacy risk is real and has regulatory implications. Under GDPR, the storage and processing of personal data must have a lawful basis. If a model's weights contain memorised personal data, that constitutes ongoing processing of that data. The GDPR right to erasure (Article 17) creates an obligation — when a data subject requests erasure of their data — that is technically difficult to satisfy when the data has been memorised into model weights.

Practical implications. Before fine-tuning on any dataset that contains personal data: apply thorough de-identification to the training corpus, use synthetic data generation to replace personal information with demographically similar but non-identifiable substitutes where possible, test the fine-tuned model for memorisation using canary strings and verbatim extraction testing, and document the legal basis for processing and the retention period for any personal data used in training.

PII Detection in Outputs

For AI systems that produce outputs served to users, PII detection in the output stream is an important control — both to prevent disclosure of third-party PII that the model may have learned or retrieved, and to detect when the model is reproducing memorised training data.

Regex-based PII detection. Regular expressions can reliably detect structured PII formats: email addresses, phone numbers (with number normalization for international formats), credit card numbers (using Luhn algorithm validation), Social Security Numbers and equivalent national identifiers, and postcode and ZIP code patterns. Regex-based detection is fast, deterministic, and has low false negative rates for well-defined PII formats. Its limitation is that it misses unstructured PII: names, addresses, and context-dependent personal details.

Named entity recognition (NER) for PII. NER models trained specifically for PII detection (spaCy, AWS Comprehend, Microsoft Presidio) can identify person names, organisation names, and location entities in free text. These are more capable than regex for unstructured PII but have higher false positive rates and are language-dependent.

Presidio is Microsoft's open-source PII detection library, widely used in AI applications because it combines regex patterns for structured PII with NER for unstructured PII and provides a configurable action pipeline (detect, anonymise, redact). Presidio can be deployed in-process or as a service and is compatible with common AI application frameworks.

Output filtering architecture. The practical architecture is: model produces output, output passes through PII detector before being served to the user, detector flags or redacts PII instances, flagged outputs are either blocked, redacted, or routed for human review depending on sensitivity level. For high-volume systems, PII detection adds latency and must be optimised for throughput.

De-identification of Training Data

When building training datasets from sources that contain PII, de-identification is the primary privacy control. De-identification aims to remove or transform identifying information so that the dataset cannot reasonably be used to identify specific individuals.

Standard de-identification approaches. Named entity removal (replacing person names, organisation names, and location names with placeholders), structured data tokenisation (replacing account numbers, phone numbers, and ID numbers with random tokens that preserve format), date shifting (shifting all dates in a document by a constant random offset to prevent re-identification via temporal context), and pseudonymisation (replacing identifiers with consistent pseudonyms that preserve referential integrity within the dataset but cannot be reversed to real identities without the pseudonymisation key).

The limits of de-identification. De-identification is not perfect anonymisation. Research has consistently demonstrated that de-identified datasets can be re-identified when correlated with other available information. The k-anonymity concept from the statistical disclosure literature — where each record is indistinguishable from at least k-1 other records on quasi-identifying attributes — provides a formal framework for evaluating de-identification strength, but achieving meaningful k-anonymity in free text is difficult.

Synthetic data as an alternative. For training data that cannot be adequately de-identified, synthetic data generation offers an alternative. A synthetic dataset contains records that match the statistical distribution of the real dataset but do not correspond to any real individual. Differential privacy-protected synthetic data generation (using techniques such as DP-GAN or DP-VAE) can provide formal privacy guarantees about the relationship between the synthetic dataset and the real individuals in the source data.

GDPR and UK GDPR Obligations for AI Systems

GDPR and UK GDPR impose specific obligations on AI systems that process personal data. Engineers building AI systems for the EU or UK market must understand these obligations because they have engineering implications.

Lawful basis for processing. Every processing activity on personal data must have one of six GDPR lawful bases (consent, contract, legal obligation, vital interests, public task, or legitimate interests). For AI systems, the lawful basis must be identified and documented before processing begins. Training a model on customer data under a "legitimate interests" basis requires a legitimate interests assessment; using customer data for a purpose that customers did not consent to when providing it likely violates GDPR.

Article 22: Automated decision-making. GDPR Article 22 gives data subjects the right not to be subject to decisions based solely on automated processing that produce significant effects concerning them. This applies to AI systems that make or substantially influence decisions about individuals — creditworthiness assessments, hiring screening, fraud flagging, medical triage. Systems covered by Article 22 must provide: the right to request human review of the decision, meaningful information about the logic of the automated processing (explainability), and the ability to contest the decision. Engineering teams building such systems must design for explainability and human review from the start.

Right to erasure (Article 17). Data subjects can request deletion of their personal data. For AI systems, this creates a challenge: if the model was trained on data that included the subject's personal information and the model has memorised it, satisfying the erasure request may require retraining the model. This is a known open problem in AI privacy. Practical approaches include: designing training pipelines with erasure in mind from the start (maintaining which training examples each data subject contributed, enabling targeted retraining that excludes erased data), using machine unlearning techniques (approximate methods for reducing the influence of specific training examples without full retraining), and evaluating whether the model's memorisation of specific data is material (if the model cannot reproduce the data on targeted extraction testing, the practical risk of the memorisation may be low).

Data minimisation. GDPR's data minimisation principle requires that only the personal data necessary for the specified purpose is processed. For AI systems, this means: do not include PII in prompts unless it is necessary for the task, do not store conversation history longer than necessary for the service function, do not log model inputs and outputs that contain PII unless there is a specific retention justification.

Confidential Information in Shared Model Contexts

A distinct category of risk arises from the use of multi-tenant AI APIs for processing confidential business information.

Consumer AI APIs. When engineers use consumer-tier AI API keys (personal OpenAI, Anthropic, or Google AI accounts without an enterprise agreement), the data submitted via the API may, by default, be used for model training and improvement. Most major providers allow enterprise customers to opt out of this data use, but consumer-tier usage typically does not have this protection. Processing confidential client information, trade secrets, or personal data under a consumer API agreement almost certainly violates data protection obligations and potentially violates client confidentiality agreements.

Enterprise AI API agreements. Enterprise agreements with AI providers typically include: data processing agreements (DPAs) required under GDPR when using a third-party processor; provisions for data not being used for model training; logging and audit rights; and sometimes specific data residency guarantees. Before using any AI API to process confidential or personal data, confirm that an appropriate enterprise agreement and DPA are in place.

Data residency requirements. Some industries and jurisdictions impose data residency requirements: personal data or specific categories of data must not leave a specified geography. For AI systems processing data subject to residency requirements, this means: the AI API endpoint must be in the required geography (many providers offer regional API endpoints), model inference must occur in the required region, and logging and storage of inputs and outputs must also be in the required region. Verify residency requirements before selecting an AI API and confirm the provider offers a compliant regional endpoint.

Self-hosted models for confidential workloads. For workloads where data cannot be sent to any third-party API — due to data residency requirements, confidentiality obligations, or regulatory restrictions — self-hosted open-weights models are the appropriate solution. The engineering and operational cost of self-hosting is higher, but it provides the strongest data protection guarantee: no data leaves your infrastructure. Open-weights models in the Llama 3, Mistral, and Gemma families cover a wide range of capability tiers at mid-2026 and are viable for many production workloads that do not require frontier model capability.