AI Hallucinations — How to Catch Them Before They Catch You
Deliberate Academy Editorial Team
Reviewed for accuracy and professional relevance
You're 3 lessons in — don't lose your progress.
Sign up free to save where you are and earn a verified certificate when you pass.
- Explain why AI systems hallucinate — technically and practically — and why confidence in output is not a reliability signal
- Apply the three-step verification standard to any AI-generated factual claim before using it in professional work
- Identify the five categories of professional work where hallucination risk is highest and extra verification is required
In 2023, a New York lawyer submitted a legal brief citing six case precedents in support of a client's position. The brief was detailed, authoritative, and professionally formatted. The judge discovered that every one of the cited cases was fictional — generated by ChatGPT, which had no access to the real cases but had produced plausible-sounding citations nonetheless. The lawyer faced sanctions. The client's case was damaged. And the incident became one of the most widely cited examples of what happens when AI hallucination goes unchecked in professional work.
This lesson is about understanding why hallucinations happen structurally, which professional contexts carry the highest risk, and exactly what to do about it.
Why AI Systems Hallucinate
Hallucination is not a bug in the conventional sense — it is a predictable consequence of how large language models work. As you learned in lesson one, LLMs generate text by predicting the next most likely token based on patterns in their training data. They are optimized to produce fluent, contextually appropriate text. They are not optimized to distinguish between things they know reliably and things they are interpolating or confabulating.
When a model encounters a question for which its training data does not contain a reliable, direct answer — a specific legal case, a precise statistic, the exact wording of a regulation — it does not say "I don't have reliable information about this." It generates the most statistically plausible continuation of the prompt. For a legal question, that continuation looks like a legal citation. For a statistical question, it looks like a number with appropriate precision. For a question about a person's professional history, it looks like a plausible career trajectory. All of these outputs read as confident and credible because confident, credible prose is what the model was trained to produce.
The confidence paradox is the feature of hallucination that makes it most dangerous professionally. You might expect that an AI system would be less confident about things it does not know. The opposite is often true. A model generating a hallucinated legal citation may express more certainty and specificity than a model accurately describing a concept it has absorbed well from training. Specificity — exact case names, precise clause numbers, quoted statistics with decimal places — is a signal of plausible-sounding text, not of verified accuracy. The more specific and authoritative an AI output sounds, the more important it is to verify the specific claims it contains.
Specificity in AI output is not a reliability signal. A model that generates a case citation with the exact court, year, and judge's name is not more likely to be correct than one that gives a general description. It is generating the level of specificity that sounds credible for that type of claim — which is not the same as having retrieved a verified source.
The Five High-Risk Categories
Hallucinations occur across all types of AI output, but the professional consequences are highest in five specific categories. Knowing these categories helps you direct your verification effort appropriately rather than treating every AI output with equal suspicion.
Legal citations and regulatory references. Case names, statute numbers, clause references, regulatory guidance documents, and the specific holdings or provisions they contain are among the highest-risk AI outputs. Models generate plausible legal citations reliably — the format is consistent and the model has absorbed enormous quantities of legal text — but the specific citations are frequently fictional or misattributed. Any legal reference in an AI output requires direct verification against the actual source before professional use.
Statistics and numerical claims. AI models generate numbers that look precise — percentages, survey results, market size figures, mortality rates, economic projections — with the same fluency as everything else they produce. These numbers may be broadly in the right range, significantly off, or entirely fabricated. Specific statistical claims always require a verifiable source before use in any professional context.
Dates and recent events. Post-cutoff events are obvious hallucination risks, but date hallucination also occurs for historical events — particularly for specific meeting dates, publication dates, decision dates, and the chronology of events in a complex narrative. When the exact date matters professionally, verify it against a primary source.
Technical specifications. Product specifications, engineering standards, dosing guidelines, software version requirements, compliance thresholds, and similar technical specifics are frequently hallucinated with plausible precision. In fields where specification errors carry safety or legal consequences — healthcare, engineering, regulated industries — every technical specification in an AI output requires verification against the original technical documentation.
People and their statements. AI models regularly attribute statements, positions, and professional histories to real people inaccurately. A named executive may be credited with a quote they never gave. A researcher may be cited as the author of a study they did not conduct. A professional's biography may contain accurate details mixed with plausible-sounding fabrications. Any claim about a specific named individual — their role, their statements, their credentials — requires verification before use.
A financial analyst uses an AI tool to research a report and the output states that a pharmaceutical company's CEO made specific public statements about projected clinical trial results at a named investor conference last quarter, including a direct quote. The analyst is about to attribute that quote in a client briefing. Which hallucination category does this claim fall into, and what does that mean for how the analyst should treat it?
Select one answer.
Catching a hallucinated citation before it reached the client
Context
A junior associate at a professional services firm used an AI tool to research precedent for a client memo advising on non-compete clause enforceability in a specific jurisdiction. The AI output identified three relevant cases by name, court, year, and summarized the key holding from each. The associate incorporated the citations into a first draft memo and sent it to the firm's senior legal counsel for review before client delivery.
Action
The senior counsel's practice was to verify every case citation directly before signing off on any client-facing document. She ran each of the three cited cases through the firm's legal database. Two of the three cases did not exist. The third existed but held the opposite of what the AI had summarized — it had actually strengthened enforceability rather than limiting it, making it unhelpful for the client's position. She sent the draft back to the associate with the three failed verification checks documented, and the associate researched the actual relevant precedent through the legal database.
Outcome
The client memo was delivered with verified, accurate citations. The incident became a training case within the firm: AI-assisted legal research is encouraged as a starting point for identifying potentially relevant areas of law, but every citation requires direct database verification before inclusion in any client-facing or court-filed document. The senior counsel noted that the AI output had been formatted and written with enough authority that a less experienced reviewer might have assumed it was accurate.
The Three-Step Verification Standard
Every AI-generated factual claim that will be used in professional work should pass through a three-step verification process before it reaches a client, a colleague, a report, or a decision-maker.
Step one: Cross-reference. Search for the specific claim using an independent source — a legal database, a primary statistical publication, a government document, a news archive, a company website. You are looking for the claim to be independently confirmed by a source you can evaluate directly. If you cannot find independent confirmation, the claim is unverified.
Step two: Source test. For cited sources — documents, cases, studies, reports, statements — go to the actual source and confirm the claim is present and accurately represented. AI tools frequently cite real sources with inaccurate summaries of what those sources actually say. The source may exist; the representation of it may not be accurate.
Step three: Domain expert check. For claims in a specialized domain where you are not yourself an expert — a legal holding in an unfamiliar jurisdiction, a clinical dosing guideline, a technical engineering specification — have the claim reviewed by a domain expert before professional use. Your ability to run steps one and two is limited when you do not have the domain knowledge to evaluate whether what you found matches the claim.
For most professional AI use, steps one and two are sufficient for routine factual claims. Step three is specifically for high-stakes claims in domains outside your expertise. The verification standard is proportionate to the risk: a statistic in a casual internal briefing requires less rigor than a regulatory citation in a client deliverable. Build your verification habit around the consequence of being wrong, not around a uniform level of suspicion.
A marketing manager uses an AI tool to prepare a competitive analysis slide deck. The AI output states: 'According to Forrester Research's 2024 Digital Experience Report, 67% of enterprise customers cite personalization as their primary vendor selection criterion.' The manager is about to include this statistic in a presentation to the board. What is the correct verification step?
Select one answer.
Exercise
Your Task
Take the last three AI outputs you used in professional work — drafts, summaries, research memos, analysis documents — and list every specific factual claim in each: statistics, citations, dates, named individuals with attributed statements, and technical specifics. Run each claim through step one of the verification standard. Record which claims you can confirm independently and which you cannot.
Success looks like
- You have identified at least one claim per output that falls into one of the five high-risk categories
- For each identified claim you have noted whether independent confirmation exists — and for which claims it does not
- You can articulate why at least one unconfirmed claim would carry professional risk if used without verification
Watch out for
- Treating the absence of obvious errors as confirmation — an AI output can be fluent and professionally formatted while containing unverifiable claims
- Only reviewing outputs that felt uncertain at the time — hallucinated claims are often in the most confident-sounding sentences
- Skipping the source test on claims where you found a matching source name — check that the source actually contains the specific claim, not just the topic
Hint
Start by scanning for numbers, proper nouns (people, organizations, case names), and dates — these are the most reliable indicators of high-risk claims regardless of how authoritative the surrounding prose sounds.
Try It: AI-Graded Practice
The exercise above is self-assessed. The exercise below is graded automatically, so you can get direct feedback on whether your verification plan actually goes beyond a credibility check to confirm the specific figure in the named source.
- Hallucination is a structural feature of how LLMs work, not a random error. Models generate the most statistically plausible continuation of a prompt — plausible and accurate are not the same thing.
- The confidence paradox: AI outputs are often more specific and authoritative-sounding when the model is generating rather than recalling. Specificity is a signal of plausible prose, not of verified accuracy.
- The five high-risk categories — legal citations, statistics, dates and events, technical specifications, and attributed statements about named individuals — require verification before any professional use.
- The three-step verification standard — cross-reference, source test, domain expert check — is proportionate to the risk: apply all three steps for high-stakes claims in specialized domains, steps one and two for routine factual claims.
- Hallucinated outputs are not identifiable by style or tone — they are formatted and written with the same authority as accurate outputs. The only reliable check is verification against independent sources.