How AI Systems Actually Work: A Compliance-Relevant Model
Deliberate Academy Editorial Team
Reviewed for accuracy and professional relevance
- Explain, in plain language, how a modern AI system generates an output, without requiring a technical or engineering background
- Identify the two core ways an AI system fails differently than traditional software, and why that distinction matters for compliance work
- Distinguish a deterministic, rule-based system from a probabilistic AI system, and explain why each requires a different compliance approach
- Apply this foundational model to recognize why "the system passed testing" does not mean the same thing for an AI system that it does for traditional software
A compliance analyst is asked to review a new AI tool that scores loan applications. Her instinct, trained on years of reviewing traditional software systems, is to ask for the test results and the code review sign-off — the standard evidence that a deterministic system works as designed. The vendor provides both. What she does not yet know to ask is a different, harder question: not "does this system work as designed," but "what does this system do when it encounters a case unlike anything in its training data, and how would we ever find out." That second question is the entire reason this course exists, and this lesson builds the foundation for asking it.
Traditional Software Follows Rules. AI Systems Learn Patterns.
Traditional software executes rules a person wrote: if this condition is true, do this specific thing. Its behavior is deterministic — given the same input, it produces the same output every time, and a competent engineer can trace exactly why. Testing this kind of system means checking whether its outputs match the rules it was explicitly told to follow.
Most AI systems relevant to compliance work today — including the large language models behind tools like ChatGPT, Claude, and Copilot, and the specialized models behind AI-powered scoring, screening, and classification tools — work differently. They are trained on large volumes of historical data to recognize statistical patterns, and they generate an output based on the closest pattern match to a new input, not a rule a person explicitly wrote. This has two consequences that define almost everything else in this course.
Consequence one: outputs are probabilistic, not deterministic. The same input can produce a slightly different output on a different run, and the system can be equally confident whether an output is correct or wrong. There is no line of code to point to and say "this is where it went wrong" the way there usually is in traditional software — the "reasoning" is distributed across the statistical patterns the model learned, which is why some AI systems are much harder to fully explain than others.
Consequence two: the system inherits whatever patterns existed in its training data — including unwanted ones. If historical data reflects a bias, a gap, or an outdated pattern, the model will tend to reproduce it, because it has no independent way to know the historical pattern was wrong or should not be repeated. This is the technical root of the bias risk covered in the next lesson, and it is also why "the system passed our test set" is a weaker compliance signal for an AI system than it is for traditional software — the test set itself may share the same blind spots as the training data.
You do not need to be able to build or code an AI system to do compliance work well in this field. You need this specific mental model: an AI system generates a statistically plausible output based on patterns in its training data, not a rule-following execution of instructions a person wrote. Almost every compliance-relevant AI risk in this course traces back to that one distinction.
A compliance analyst is reviewing a new AI-powered candidate-screening tool. The vendor reports that the tool 'passed all internal testing with a 95% accuracy rate against our test set.' Based on this lesson, what is the most important limitation of this evidence?
Select one answer.
Why This Distinction Matters for Compliance Specifically
Once you hold this model clearly, several things that otherwise sound like isolated technical details start to connect: why a regulator is more interested in an AI system's training data and testing methodology than in its code; why "the AI made a mistake" is treated differently in emerging regulation than "the software had a bug"; and why human oversight — a person who can catch and correct an AI system's output — is such a consistently required control across every framework this course introduces, rather than an optional nicety.
A Compliance Review That Asked the Traditional-Software Question — Consumer Lending Fintech
Context
A compliance analyst reviewing a new AI-powered underwriting tool requested the vendor's code documentation and test pass rate, the standard evidence her team used for reviewing any new software system.
Action
A colleague newer to compliance but with a data science background asked a different question: what was the training data's demographic composition, and had anyone checked whether the model's approval rate varied meaningfully across demographic groups in a way the training data did not explain? The original review had not asked this question at all, because it does not map onto the traditional software review checklist the team had always used.
Outcome
The subsequent analysis found a modest but statistically real disparity in approval rates across two demographic groups, traceable to an underrepresentation in the historical training data rather than any explicit rule in the system. The team paused the rollout, required the vendor to retrain on a more representative data set, and revised their standard AI vendor review checklist to include training-data composition as a required question — not just code and test-pass-rate documentation.
Why does this lesson argue that human oversight is such a consistently required control across nearly every AI governance framework, rather than an optional best practice?
Select one answer.
Exercise
Your Task
Think of an AI system your organization uses or is evaluating (or a plausible one). Write two questions you would ask about it: one question in the style of a traditional software review (about rules, code, or test results), and one question in the style this lesson introduces (about training data patterns, probabilistic behavior, or how the system would be caught if it produced a confidently wrong output on an unfamiliar case). Compare the two — which question would a standard IT security review have been more likely to ask?
Your reflection
Did you complete this exercise? What did you find? (Saved locally in your browser)
- Traditional software follows explicit rules and is deterministic; most modern AI systems learn statistical patterns from training data and produce probabilistic, sometimes confidently wrong, outputs.
- An AI system inherits whatever patterns exist in its training data, including unwanted ones such as historical bias — with no independent way to know a pattern should not be repeated.
- Passing internal testing is weaker evidence for an AI system than for traditional software, because the test set can share the same blind spots as the training data.
- This distinction is the technical root of why AI governance frameworks consistently require human oversight, training-data scrutiny, and ongoing monitoring — controls that traditional software review checklists do not typically emphasize.
- You do not need to code or build AI systems to do this work well — you need the specific mental model of pattern-based, probabilistic output versus rule-following execution.