AI for Software Engineers Capstone Exercise — AI for Software Engineers

What you'll learn

Apply skills from across this course in a single realistic engineering scenario
Produce concrete, role-relevant deliverables that demonstrate AI engineering judgment across scoping, evaluation, build strategy, and stakeholder communication
Self-assess your output against professional quality criteria for each deliverable

Across this course you have worked through the engineering mental model for LLMs, the AI tool ecosystem, AI feature scoping, LLM integration patterns, product requirements and testing, stakeholder communication, team governance, and build vs. buy strategy. Each of those skills exists to serve a single outcome: engineering teams that can scope, build, evaluate, and govern AI-powered systems reliably.

This capstone puts them together in a scenario with the time pressure, ambiguity, and stakeholder dynamics that real AI engineering decisions involve.

Capstone Exercise

Feasibility Assessment and Build Recommendation for an AI-Powered Customer Support Feature

Context

You are the technical lead at a 20-person B2B SaaS company. The product is a project management tool used by operations teams. The customer success team handles approximately 200 support tickets per week using two senior support agents and four tier-1 agents. Tier-1 agents handle routine questions (how-to, billing, basic account issues). Senior agents handle escalations, bug reports, and enterprise account queries. The CEO has been demoing an AI-powered support feature informally to a prospect: the prospect saw a prototype where incoming tickets were automatically triaged into categories, a draft first response was generated for tier-1 tickets, and complex tickets were flagged for senior agent review. The prospect expressed strong interest. The CEO has asked engineering to assess feasibility, scope the feature properly, and present a build recommendation to the board in two weeks. Engineering has not yet started.

Your Task

Produce four deliverables. First, an AI feature scoping document covering: the precise capability definition for each of the three sub-features (triage, draft response, escalation routing), an accuracy requirement for each sub-feature with separate precision and recall thresholds, the failure mode specification for each sub-feature (what happens when the AI is wrong), and the inference-time data requirements. Second, a minimal evaluation set design for the triage and drafting sub-features: how many examples, what categories, how will ground truth be determined, and what quality dimensions will be measured for the drafting feature. Third, a build vs. buy recommendation covering the infrastructure approach, data sensitivity assessment for customer support ticket content, estimated monthly cost at 200 tickets per week with projected cost at 1,000 tickets per week, and model provider portability recommendation. Fourth, a two-paragraph board summary that explains the feature's AI trade-offs in plain language, covers what happens when the AI is wrong, and gives the board a clear recommendation on whether to proceed and under what conditions.

Your notes (optional)

Deliverable

Four documents: (1) AI feature scoping document with capability definitions, accuracy requirements, failure mode specs, and data requirements for all three sub-features; (2) evaluation set design for triage and drafting sub-features including example counts, categories, ground truth method, and quality dimensions; (3) build vs. buy recommendation with infrastructure choice, data sensitivity assessment, cost estimate at 200 and 1,000 tickets per week, and portability recommendation; (4) two-paragraph board summary explaining AI trade-offs in plain language, failure mode consequences, and a clear proceed/do not proceed recommendation with stated conditions.

Quick check

The third capstone deliverable pairs a data sensitivity assessment with an infrastructure choice. What ordering between the two does the brief assume?

Select one answer.

Key takeaways

AI feature scoping requires capability definitions precise enough to derive measurable accuracy requirements. Vague capability definitions produce vague acceptance criteria and unmeasurable done conditions.
Evaluation set design must address both the task type (classification vs. generation) and the specific quality dimensions that matter for the business use case. Triage and drafting require different evaluation approaches.
Build vs. buy decisions in AI require data sensitivity assessment as a first-order constraint, not an afterthought. Infrastructure choice follows from data requirements, not the other way around.
Board-level AI communication is most credible when it leads with failure modes and quantifies their frequency. A board that understands what happens when the AI is wrong, and how often, can make a better investment decision than one that has only seen the success case.