AI Security Engineering Capstone Exercise

This capstone integrates the security concepts from all nine lessons of the course into a single realistic scenario. The system described below is the kind of AI deployment that is genuinely common in mid-2026 — and the kind that consistently surfaces multiple serious security gaps in penetration testing and compliance reviews.

You will produce four deliverables for this system. Each deliverable is scaffolded with a description of what it should cover, followed by self-assessment criteria that let you evaluate the completeness and quality of your own work.

There are no automated graders for this exercise. The value is in the depth of thinking you apply. A capstone that takes 20 minutes and produces thin answers has not developed the judgment that the exam and the job both require. A capstone that takes 90 minutes and produces specific, technically grounded analysis builds exactly the kind of reasoning that distinguishes an AI security practitioner from someone who has merely read about the topic.

The System

Organisation: Veritas Commerce, a mid-market e-commerce operator with 1.2 million registered customers across the UK and EU.

System: A customer support chatbot deployed on the public-facing website and mobile app, accessible without authentication for pre-sales queries and with customer account authentication for post-purchase queries.

Architecture:

• Model: Claude 3.5 Sonnet (Anthropic API, enterprise agreement), with a detailed system prompt that defines the assistant's role, allowed topics, and escalation paths
• RAG corpus: Two corpora retrieved from a shared vector database (Pinecone):
  • A product knowledge base (~80,000 chunks) authored by the Veritas content team
  • A customer support history corpus (~200,000 chunks) containing historical support conversations, including resolved tickets that include customer names, order details, and account information
• Tool integrations:
  • get_order_status(order_id: str) — queries the order management system for the authenticated customer's own orders
  • create_support_ticket(category: str, description: str, priority: str) — opens a support ticket in the CRM system
  • initiate_return(order_id: str, item_id: str, reason: str) — initiates the return and refund flow for the authenticated customer
• Authentication: Unauthenticated users can access the chatbot for product questions; authenticated users (logged in with customer account) additionally have access to the get_order_status and initiate_return tools
• Public web interface: The chatbot iframe is embedded in public pages served to any visitor globally
• Fine-tuning: None — uses the base Claude 3.5 Sonnet model with a custom system prompt
• Logging: Basic application logs capturing session IDs, timestamps, and error codes — no logging of prompt content or model outputs

Known context:

• The support history corpus has not been reviewed for PII since it was ingested 18 months ago. It was sourced directly from the CRM export and was not de-identified.
• The system prompt contains detailed escalation instructions including the names and email addresses of specific customer service team members.
• The product knowledge base is updated weekly via an automated pipeline that fetches content from a third-party content management system. The CMS is accessible to any Veritas employee with a company email address.
• Veritas operates a returns fraud detection program. A high rate of return initiations from a single session is flagged for human review. The chatbot's initiate_return tool feeds directly into this flow.
• Veritas is expanding into the EU market. Its customer base in Germany and France is growing rapidly, and its AI systems are now in scope for EU AI Act assessment.

---

Deliverable 1: STRIDE Threat Model

Produce a threat model for the Veritas customer support system using the STRIDE framework.

What to cover:

For each of the six STRIDE categories (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege), identify:

• At least one specific threat instance relevant to this architecture
• Which component or data flow the threat targets
• The severity assessment (considering both likelihood and impact in this specific context)
• Whether the threat is AI-specific, traditional software, or a hybrid

Additionally, identify the top three threats across all STRIDE categories that you assess as the highest combined severity for this specific system and context. Justify your prioritisation.

Beyond STRIDE, identify two threats from the OWASP LLM Top 10 that STRIDE alone would likely under-represent for this architecture. Explain why STRIDE misses them and how you would surface them in the threat model.

Self-assessment criteria:

Your threat model should be evaluated against the following:

• Have you identified the indirect prompt injection surface created by the product knowledge base pipeline? The CMS is editable by any Veritas employee — this is an indirect injection surface that any employee (or anyone who compromises a Veritas employee's account) can exploit.
• Have you identified the customer support history corpus as both an indirect injection surface (historical conversations can be surfaced by retrieval) and a PII leakage surface (the corpus contains customer personal data)?
• Have you identified the initiate_return tool as a high-consequence agentic action that makes prompt injection an operational security incident rather than a content quality issue?
• Have you identified repudiation as a significant threat given the current logging architecture (session IDs only, no content logging)?
• Have you correctly assessed the STRIDE Tampering category to include RAG corpus poisoning via the CMS pipeline, not just traditional data tampering?
• Have you identified LLM02 (Sensitive Information Disclosure) and LLM06 (Excessive Agency) as the two OWASP categories most likely to be under-represented by a STRIDE analysis focused on the traditional software components?

---

Deliverable 2: Prompt Injection Defence Architecture

Design the injection defence architecture for the Veritas chatbot system.

What to cover:

For each of the five defence layers, specify the controls you would implement for this system:

Layer 1: Input controls. What validation, sanitisation, or blocking would you apply to the user's message before it reaches the model? Address both the unauthenticated and authenticated user surfaces separately. What would you block entirely versus flag for additional scrutiny?

Layer 2: Context isolation. How would you structure the model's context to separate the system prompt, the retrieved product knowledge base content, the retrieved customer support history content, and the user's message? Which XML tags or structural patterns would you use? How would you label each context section for the model?

Layer 3: Privilege architecture. The initiate_return and get_order_status tools are only available to authenticated users. But authentication alone is not sufficient privilege separation for an agentic system. Describe the additional privilege constraints you would add: which tools should be available in which conditions, what parameter validation the tool execution layer should perform independently of the model's reasoning, and what confirmation or approval gates (if any) you would add before high-consequence tool calls.

Layer 4: Output controls. What would you validate in the model's response before serving it to the user? What would you validate before executing any tool call the model requests? Be specific about what schema or content validation you would apply and what happens when validation fails.

Layer 5: Monitoring and detection. What would you log that the current system does not log? What anomaly signals would you monitor? At what thresholds would you alert, and what would the alert trigger?

Additionally, address the CMS pipeline injection risk specifically: the product knowledge base is updated weekly from a CMS editable by any Veritas employee. What controls would you add to the update pipeline to reduce the risk of injected content reaching the RAG corpus and being retrieved during customer interactions?

Self-assessment criteria:

Your defence architecture should be evaluated against the following:

• Have you specified different privilege controls for the initiate_return tool (high consequence) versus create_support_ticket (low consequence)? A return initiation has financial impact; a ticket creation does not. These should have different approval gates.
• Have you addressed the tool parameter validation layer independently of the model? The model should not be the only thing preventing initiate_return(order_id="all", item_id="all", reason="refund everything"). The tool execution layer should independently validate that the order_id belongs to the authenticated session's customer.
• Have you specified full content logging of inputs, retrieved context, and model outputs — not just metadata — and addressed the PII-in-logs implication?
• Have you addressed the CMS pipeline with controls that go beyond "review the knowledge base periodically"? Specific controls: approval gate before new content is indexed, automated injection pattern scanning on incoming CMS content, and sandboxed RAG testing before promoting updated corpus to production.
• Have you distinguished between unauthenticated injection risk (lower — no tool access to financial actions) and authenticated injection risk (higher — tool access to return initiation)?

---

Deliverable 3: PII Handling and Data Governance Plan

Design the PII handling and data governance plan for the Veritas chatbot system.

What to cover:

PII pathway mapping. Map each PII pathway to the controls you would implement:

• Pathway 1 (user input): Authenticated users may mention their name, address, payment details, and the details of third parties (gift recipients, etc.) in free-form messages.
• Pathway 2 (RAG retrieval): The customer support history corpus contains historical customer names, order details, contact information, and in some cases sensitive complaint content.
• Pathway 3 (conversation history): The chatbot maintains conversation context across the session. An authenticated user who mentions personal details early in a conversation may have those details in the model's context 15 turns later.
• Pathway 4 (the PII-in-logs problem): You have specified full content logging in Deliverable 2. Those logs contain everything in pathways 1-3.

For each pathway, specify: what PII is expected, the primary risk (leakage to the user, leakage to third parties, inappropriate retention, or regulatory exposure), and the control you would implement.

De-identification of the support history corpus. The corpus was ingested 18 months ago without de-identification and is now embedded in the vector database. What are your options for remediating this? Evaluate at least two approaches (re-ingestion with de-identification, or customer data deletion on request with targeted corpus management) including the cost and completeness of each.

GDPR obligations. Address the following specific obligations:

• Lawful basis: On what basis is Veritas processing the personal data customers share with the chatbot? Is the existing customer contract sufficient, or is a specific consent mechanism required for AI processing?
• Article 22: Does the chatbot's initiate_return tool trigger Article 22? The tool takes an automated action (initiating a return) based on the customer's request processed by an AI system. Does this constitute automated decision-making with significant legal effects?
• Right to erasure: A customer requests deletion of all their personal data under GDPR Article 17. What does Veritas need to do, specifically, to satisfy this request given that the customer's data may exist in: the support history RAG corpus, the content logs from previous chatbot sessions, and potentially in any future fine-tuning dataset?
• Data residency: Veritas has customers in Germany and France. The Anthropic API processes inference in Anthropic's data centres. What are the data residency implications, and what does Veritas need to verify about its Anthropic enterprise agreement?

Self-assessment criteria:

Your PII plan should be evaluated against the following:

• Have you correctly identified the support history corpus as requiring urgent remediation rather than deferred action? A de-identified RAG corpus is both a regulatory requirement and a significant reduction in the indirect injection risk surface from Deliverable 2.
• Have you addressed the Article 22 question accurately? The initiate_return tool, in this system, does not make a decision with significant legal effects — the customer is requesting the return, and the tool is executing their instruction. Article 22 would be triggered if the chatbot were making decisions about whether the customer was entitled to a return (eligibility decisions), not merely initiating a process the customer has asked for. Many engineers incorrectly apply Article 22 to any AI interaction involving a customer action.
• Have you identified that the support ticket PII-in-logs problem requires GDPR-compliant retention policies (not indefinite storage), access controls restricted to roles that need the data, and a documented deletion process when customers exercise their erasure right?
• Have you identified that Anthropic's enterprise agreement should include a Data Processing Agreement (DPA) as required under GDPR Article 28 for processors, and that Veritas should verify that Anthropic's EU API endpoints process data within the EU or that appropriate transfer mechanisms (Standard Contractual Clauses) are in place for transatlantic data transfer?

---

Deliverable 4: EU AI Act Compliance Assessment

Produce an EU AI Act compliance assessment for the Veritas chatbot system.

What to cover:

Classification. Classify the Veritas chatbot under the EU AI Act risk tiers:

• Is the system prohibited under Article 5?
• Is the system high-risk under Annex III? Review the relevant Annex III categories carefully. Note that the system processes customer support queries, initiates returns, and opens support tickets. Which categories might apply? Which do not?
• Does the initiate_return tool change the classification? Does processing financial actions (returns, refunds) place the system in the Annex III essential private services category?
• What is the correct classification, and what are the primary compliance obligations that arise from it?

Transparency obligations. Regardless of the system's risk classification, what transparency obligations apply under the EU AI Act? The chatbot presents itself as a customer service agent. Are there specific disclosure requirements about its AI nature?

Gap analysis. Based on the correct classification and current system description, identify the compliance gaps. Evaluate the current system against each applicable obligation and state clearly: compliant, non-compliant, or unable to assess with current information.

Prioritised engineering checklist. Produce a list of the five most important engineering changes required for compliance, ordered by risk of non-compliance rather than implementation effort. For each item, state: what the current gap is, what change is required, and which specific EU AI Act article the change satisfies.

Self-assessment criteria:

Your compliance assessment should be evaluated against the following:

• Have you correctly classified the system? The Veritas chatbot is most likely minimal-risk under the EU AI Act. The initiate_return tool does not make credit, insurance, or social benefits decisions about the customer — it executes a process the customer has requested. The Annex III essential private services category covers AI that evaluates or denies access to services, not AI that facilitates customer-requested service processes. A returns chatbot is analogous to a self-service kiosk, not a creditworthiness assessor.
• However, if Veritas were to add a feature that assessed whether a customer's return request was eligible for a refund (eligibility scoring) rather than simply initiating the process, the classification could change. Have you noted this important boundary?
• Have you correctly identified the Article 50 transparency obligation (informing users they are interacting with an AI) as the primary EU AI Act obligation for a minimal-risk system?
• Have you identified that even for a minimal-risk classification, GDPR obligations still apply in full — and that many of the engineering changes from Deliverables 2 and 3 (logging, PII controls, human oversight for complaints) are required by GDPR even if not required by the EU AI Act?
• Have you noted that if the system were to be used by Veritas to assist in decisions about customer accounts (fraud scoring, account suspension recommendations, preferential service allocation), the classification would likely change to high-risk under Annex III?

---

Bringing It Together

The four deliverables above are not independent — they share components and reinforce each other. A well-completed capstone will show how the decisions in one deliverable constrain or enable decisions in another:

• The injection defence architecture (Deliverable 2) directly reduces the PII leakage risk from RAG retrieval (Deliverable 3) by preventing injected instructions from causing the model to surface other customers' data.
• The full content logging you specified in Deliverable 2 creates the PII-in-logs obligation you address in Deliverable 3.
• The EU AI Act classification (Deliverable 4) determines whether the logging requirements in your architecture need to meet Article 12's structured logging standard or just your own operational requirements.
• The de-identification of the support history corpus (Deliverable 3) reduces both the GDPR exposure and the indirect injection surface identified in the STRIDE model (Deliverable 1).

When you review your completed deliverables, check that these connections are visible in your reasoning. The mark of a security practitioner who has genuinely internalised these concepts is the ability to see how a change in one dimension of the system affects the security posture across all the others.