Model hallucinations: why AI confidently says things that are not true

Why AI confidently says things that are not true, how often it happens and what to do so it does not catch you out. Six techniques with ready-made prompts, a checking checklist and 17 linked sources.


A reference sheet from AI Sandbox Lab for people in the practical groups.

  • Who it is for: Product Managers, Project Managers, business analysts, QA, Engineering Managers, and anyone who works with AI without a coding background
  • Reading time: about 15 minutes
  • Sources: studies and documentation, links checked on November 2, 2026; the numbers in square brackets lead to the list at the end

Map of this sheet

Map of the sheet: six topics about hallucinations and the numbers of the sections where they are covered
Fig. 1. Map of the sheet: six topics about hallucinations and the numbers of the sections where they are covered

The short version: smooth text is not a fact

Sometimes a model produces plausible untruth: an exact figure that does not exist, a link to an article that was never written, a quote nobody said. This is not a defect of one particular service. It follows from how models are built and trained. You cannot remove it completely, but you can cut the risk a lot: give the model your document, allow it to say "I don't know", require quotes, and check the important parts by hand. The more an error costs, the stricter the check.


1. What a hallucination is

A model hallucination is a plausible, confidently delivered answer that does not match reality or the source data. Researchers describe it in a similar way: the model produces "plausible but nonfactual" text [3].

The word is not quite right. For a person, a hallucination is perceiving something that is not there. A model has neither perception nor intent to lie. The authors of one of the key papers note directly that this phenomenon is "fundamentally different" from the human experience [1]. So from here on we will say it more simply: the model produced plausible untruth.

Useful Everyday analogy: the straight-A student at the board

Picture a student who has learned how correct answers sound but does not remember the facts. They have been called to the board, and they know that staying silent is not an option: a blank answer is graded worse than an attempt. They speak smoothly, in a confident tone, in the right terminology. Sometimes they hit the mark, sometimes not, and you cannot tell from the intonation. A model works in a similar way: it continues text the way such text usually sounds, rather than checking a database of facts.

The model continues text: on the left the expectation (looking up a fact in a database), on the right the reality (picking the most plausible word)
Fig. 2. The model continues text: on the left the expectation (looking up a fact in a database), on the right the reality (picking the most plausible word)

2. Why the model does this

In short, three reasons.

1. The model learns to predict text, not to store facts. It has seen a huge amount of text and learned to continue it plausibly. If a fact appeared in training rarely or only once, the model knows it poorly. The theoretical work by Kalai and Vempala shows that even with perfect training data, the error rate on such rare facts is roughly equal to the share of facts seen exactly once in training [4]. This is a mathematical result for an idealized model, not a direct measurement on a specific product.

2. Training and evaluations reward guessing. This is the main conclusion of a recent paper by OpenAI researchers, "Why Language Models Hallucinate" (2025). Most of the tests used to compare models give 1 point for a correct answer and 0 for "I don't know". Then a model that always guesses scores higher than a model that honestly admits it does not know. The authors write that training and evaluation "reward guessing over acknowledging uncertainty" [1]. It is like a student on an exam with no penalty for wrong answers: it pays to write something.

The test rewards guessing: a model that always guesses scores more points than an honest one and makes more mistakes
Fig. 3. The test rewards guessing: a model that always guesses scores more points than an honest one and makes more mistakes

3. The model is bad at judging its own confidence. In the SimpleQA test (4,326 short factual questions), the authors asked the model to state its confidence in an answer. The finding: the model's confidence is related to accuracy, but models "consistently overstate" it [5].

Useful Two words you will meet in the research

Calibration is how closely an answer's confidence matches its real accuracy. A well-calibrated model that says "I am 70% sure" is right about 70% of the time. A benchmark is a standard set of questions used to compare models, like an exam.


3. Kinds of hallucinations

Scientific surveys most often use two ways of dividing them.

First: by relation to what you gave the model. This is the classification from the survey by Ji and co-authors [2].

  • Contradicts your source. You gave a document, and the model distorted it in a summary. In the survey the example is that the text says a vaccine was approved in 2019, and the summary says 2021.
  • Cannot be checked against your source. The model added something that is not in the document. The authors note that such an addition is not always wrong, but it calls for caution, because it cannot be verified against the source.

Second: an error about the world or an error about your task. The survey by Huang and co-authors divides hallucinations into factual ones (they differ from real facts) and "faithfulness" ones (they differ from your input or instruction) [3].

Three kinds of errors: meeting minutes and three wrong retellings
Fig. 4. Three kinds of errors: meeting minutes and three wrong retellings

What this looks like at work:

RoleExampleKind
Product ManagerYou ask to condense 20 customer interviews. The summary contains a quote nobody saidnot from the source
Product ManagerYou ask for market size data. You get an exact figure with a reference to a report that does not existfactual error
Project ManagerYou ask to retell the meeting minutes. A deadline has shifted in the retelling and the owner is wrongcontradicts the source
Business analystYou ask to pull requirements out of a regulation. A clause appears that is not in the regulationnot from the source
QAYou ask for test cases from a feature description. Some cases check behavior nobody promisednot from the source
Engineering ManagerYou ask for links to articles about team assessment practices. Some titles and authors are made upfactual error

Important

The most dangerous answer is not an absurd one but a plausible one: the right format, a confident tone, real names next to invented details. In a study of made-up bibliographic references, some of the references to real works contained substantial errors in the details (43% for GPT-3.5 and 24% for GPT-4), that is, a mix of true and false [8].


4. How common it is

There is no single "the model lies in N% of cases" figure. The rate depends on the task, the model, the way of checking, and the date. Below are examples of measurements. You cannot average them or compare them with each other, because the tasks differ.

How many errors: the error rate in five studies, from 2-3% when summarizing a document to 55% of invented references
Fig. 5. How many errors: the error rate in five studies, from 2-3% when summarizing a document to 55% of invented references

Summarizing a document (the mildest condition). The Vectara leaderboard measures how often a model adds to a short summary of a document something that is not in it. As of October 22, 2026, the best models in the table are at about 2-3%, a number of well-known models at 4-12%, and one model from the GPT-5 line at 15.1% [6]. Caveats: this only covers retelling short texts, the set of documents is closed, and the scoring is done by another model, not by a person.

Factual questions without a source. On the PersonQA test (questions about public facts about people), according to the developer, OpenAI's o3 model was wrong in 33% of answers, o4-mini in 48%, and the earlier o1 in 16% [7]. The report is dated April 2025. This is the vendor's internal test.

References to literature. In the work by Walters and Wilder (2023), ChatGPT wrote short reviews on 42 topics, 636 references in total. 55% of the references from GPT-3.5 and 18% from GPT-4 turned out to be invented [8]. A recent small study in dentistry (15 questions, 2025) found a high share of invented references in GPT-4o, while systems that search real publications had no invented references but did misinterpret what they found [9]. The sample is small: take it as an illustration, not as an estimate.

Specialized legal AI tools. A Stanford team tested commercial legal research services sold as "hallucination-free". They were wrong in 17-33% of queries [10]. This is 2024 work, and the product versions have changed since.

An important caveat for conclusions. Newer models do not necessarily hallucinate less. The OpenAI system card says that o3 makes more claims, so it has more correct and more incorrect ones, and the researchers do not yet know the reason [7]. On the Vectara leaderboard, gpt-5-high has 15.1% and gpt-4o has 9.6%, so the newer model turned out worse on this particular task [6]. The models have different versions and dates, so this is an observation, not a law. The conclusion: "the model is new and smart" does not replace checking.

Useful Can it be brought down to zero

According to the theoretical work by Xu and co-authors, it is impossible to eliminate hallucinations in language models completely [11]. This is a mathematical statement under certain assumptions, but the practical conclusion matches Anthropic's recommendation in its documentation: the techniques reduce hallucinations significantly but do not eliminate them completely, and important information should be verified [14]. Plan a process with checking instead of hoping for a perfect model.


5. Well-known real cases

Lawyers and invented cases (USA, 2023). In Mata v. Avianca, a lawyer used ChatGPT to find court precedents and filed a document with cases that do not exist. On June 22, 2023, Judge Castel ordered two lawyers and their firm to pay a $5,000 fine and to notify the judges whose names had been listed as authors of the fictitious decisions. The court found that the lawyers acted in bad faith: after the problem came to light, they stood by the invented cases [12]. The lesson is not the error itself but that the answer was not checked and then was defended.

An airline chatbot (Canada, 2024). The Air Canada chatbot told a passenger that a bereavement discount on a ticket could be claimed after the trip. The real rules did not allow that. In February 2024 a tribunal in British Columbia held the airline responsible for what its chatbot said. The amount in dispute was small (about 880 Canadian dollars), but the decision became an example: a company is responsible for the answers of its bot [13]. The details come from a secondary source; the link to the decision itself is in the list.

Legal AI services. The Stanford study (see above) showed that even paid professional tools with database search are wrong noticeably more often than the sellers promise [10].


6. How to reduce the risk

All the techniques share one idea: do not ask the model to recall, give it material and require it to rely on that material. The approach in which the model first retrieves the needed documents and then answers is called RAG (retrieval plus generation) in the literature. It appeared in the work by Lewis and co-authors (2020), where this scheme gave more accurate and more factual texts than a model without retrieval [15]. RAG does not cancel errors: in the legal study [10], the tools with retrieval were still wrong, and in the dental study [9], the systems with retrieval did not invent references but misinterpreted what they found.

A closed book and an open book: an answer from memory against an answer from your document with quotes
Fig. 6. A closed book and an open book: an answer from memory against an answer from your document with quotes

Practice Technique 1. Provide a document and limit the source

A recommendation from Anthropic's documentation: ask the model to use only the provided documents rather than general knowledge [14].

Answer using only the document below. Do not use general knowledge.
If the document does not contain the answer, write: "The document does not say".
<document>
(paste the text)
</document>
Question: (your question)

Practice Technique 2. Allow it to say "I don't know"

Anthropic writes that explicit permission to admit uncertainty can sharply reduce the number of false statements [14]. This also ties to the reason from section 2: by default, the model feels obliged to answer.

If you are not sure of an answer or lack data, say so:
"Not sure" or "Not enough data". Do not make things up.
List separately what you know for certain in the answer and what you are assuming.

Practice Technique 3. Quotes first, conclusions after

For long documents, Anthropic advises first asking for word-for-word quotes and drawing conclusions only from them [14].

First pull out word-for-word quotes from the document that relate to the question, with numbers.
If there are no suitable quotes, write "No quotes found".
Then draw a conclusion, referring to the quote numbers. Add nothing beyond the quotes.

Practice Technique 4. Let the model check its own draft

The Chain-of-Verification method (Dhuliawala and co-authors, 2023): the model writes a draft, formulates verification questions, answers them separately from the draft, and produces a corrected final version. The authors report that it reduces hallucinations in several tests [16]. This is a preprint, not a peer-reviewed paper. Below is a simplified version for a chat.

You wrote the answer above. Now:
1. List all verifiable claims (figures, dates, names, links).
2. For each, formulate a check question and answer it without looking at the draft.
3. Compare. Remove or flag whatever was not confirmed.
4. Give the corrected answer.

A caveat: the model is checking itself, so this does not always help. Do the main checks (figures, links, legal and financial facts) against the primary source.

Practice Technique 5. Ask the question several times

The idea comes from the SelfCheckGPT method: if the model knows the topic, repeated answers agree, while invented details change from time to time [17]. Anthropic recommends the same technique: run one request several times and compare, since differences may point to a hallucination [14]. Ask one question in three fresh chats. Check by hand everything that did not match. Agreement does not prove correctness: a model can consistently repeat the same error.

Practice Technique 6. For agents in an IDE: the source in the project folder

If you work with an agent (Cursor, Claude Code, Antigravity), put the needed documents in the project folder and add this line to the rules (CLAUDE.md or AGENTS.md):

Treat as facts only what is in the project files.
If a fact is not in the files, write "needs clarification" instead of inventing it.
For every important conclusion, add the file name and a quote.

7. Checklist: how to check a model's answer

Checking is not always needed. The higher the cost of an error, the stricter the check.

Checking by cost of error: three levels and what to check at each
Fig. 7. Checking by cost of error: three levels and what to check at each
  1. The cost of an error. Is this a draft for yourself, an email to a client, or a document for lawyers and accountants? The higher the cost, the more checking.
  2. Is there a source? Did you ask the model to rely on your document? If not, treat the answer as a hypothesis.
  3. Figures, dates, names. Pick the 3-5 most important and check them against the primary source. If they all match, the chance of an error is lower, but not zero.
  4. Links and quotes. Open every link. Make sure the page exists and says what the model attributes to it. A real article title with an invented conclusion is a common pattern.
  5. Word-for-word quotes. Find the quote in the document with a text search (Ctrl+F). If it is not in the document, it was invented.
  6. The answer is too smooth. No caveats, no "I don't know", it fits your request perfectly. That is a reason to be wary.
  7. Repeat the question. Do the answers differ in details? Check exactly those details.
  8. Ask "where does this come from?" Ask the model to name the source and a quote. If it cannot, there is nothing behind it. But remember: a model can also produce an invented source, so open it yourself.
  9. A second person. For important decisions, have a colleague look at the conclusions and the sources.
  10. Write down how you checked. One line in the document: "figures checked against report X, links opened". That way you can repeat the check.

Important

Responsibility for the result stays with the person. In both well-known cases above, the blame fell not on the model but on those who used its answer without checking, or who released a bot without control [12][13].


What to take away

  • A hallucination is not a glitch but a consequence of how the model is built and how it is evaluated [1][4].
  • The rate depends on the task: summarizing a document gives single-digit percentages, while questions about rare facts and references to literature give tens of percent [6][7][8].
  • A new model does not guarantee fewer errors [6][7].
  • The best way to reduce risk: give a document, allow "I don't know", require quotes, and check the main points by hand [14][15].

Appendix A. Glossary

TermIn plain words
Model, language modelA program that learned to continue text from a huge number of examples. ChatGPT, Claude and Gemini run on such models.
HallucinationA plausible, confident answer that does not match the facts or your document.
PromptThe text of a request that you send to the model.
Source, documentMaterial you give the model: minutes, a regulation, a report, a table. The answer is then checked against it.
CalibrationHow closely the model's confidence matches its real accuracy. Like a weather forecast: if "70% chance of rain" comes true in 7 cases out of 10, the forecast is calibrated.
BenchmarkA standard set of questions used to compare models. Like a common exam for everyone.
RAG (retrieval plus generation)A scheme in which the model first finds the needed documents and then answers from them. An open-book exam.
Chain-of-VerificationA self-checking technique: draft, verification questions, answers to them, corrected final version (section 6, technique 4).
SelfCheckGPTA way to catch invention by inconsistencies: the same question is asked several times and the answers are compared (section 6, technique 5).
System cardA report from the model's developer on its capabilities, tests and risks. Written by the vendor itself.
PreprintA scientific paper posted before review by other scientists. Useful, but less thoroughly checked than a journal paper.
AgentA model that does not only answer in a chat but carries out steps itself: reads files, searches, edits documents.
CLAUDE.md, AGENTS.mdA project rules file that the agent reads at the start of work. A handy place for the rule "facts only from the files".

Appendix B. Sources

Links checked on November 2, 2026.

  1. Kalai A. T., Nachum O., Vempala S. S., Zhang E. Why Language Models Hallucinate. 2025. arXiv:2509.04664 (preprint). https://arxiv.org/abs/2509.04664
  2. Ji Z., Lee N., Frieske R. et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, vol. 55, no. 12, 2023. https://doi.org/10.1145/3571730 (preprint: https://arxiv.org/abs/2202.03629)
  3. Huang L., Yu W., Ma W. et al. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems, 2024/2025. https://doi.org/10.1145/3703155 (preprint: https://arxiv.org/abs/2311.05232)
  4. Kalai A. T., Vempala S. S. Calibrated Language Models Must Hallucinate. STOC 2024 (preprint 2023). https://arxiv.org/abs/2311.14648
  5. Wei J., Karina N., Chung H. W. et al. Measuring short-form factuality in large language models (SimpleQA). OpenAI, 2024. arXiv:2411.04368 (preprint). https://arxiv.org/abs/2411.04368
  6. Vectara. Hallucination Leaderboard (HHEM). A live leaderboard, last updated 22.10.2026 at the time of checking. https://github.com/vectara/hallucination-leaderboard
  7. OpenAI. OpenAI o3 and o4-mini System Card. April 16, 2025. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf
  8. Walters W. H., Wilder E. I. Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13, 14045, 2023. https://doi.org/10.1038/s41598-023-41032-5
  9. Hooshiar M. H. Artificial intelligence reliability in implant dentistry: a comparative analysis of clinical accuracy and hallucination patterns across multiple language models. Journal of Prosthetic Dentistry, 135(5), e155-e161, 2025. https://doi.org/10.1016/j.prosdent.2025.11.005 (per PubMed, abstract only)
  10. Magesh V., Surani F., Dahl M., Suzgun M., Manning C. D., Ho D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. 2024. arXiv:2405.20362 (preprint). https://arxiv.org/abs/2405.20362
  11. Xu Z., Jain S., Kankanhalli M. Hallucination is Inevitable: An Innate Limitation of Large Language Models. 2024. arXiv:2401.11817 (preprint). https://arxiv.org/abs/2401.11817
  12. Mata v. Avianca, Inc., No. 22-cv-1461, S.D.N.Y., sanctions decision of 22.06.2023 (Judge P. K. Castel). Summary: ABA Journal, "Lawyers who doubled down and defended ChatGPT's fake cases must pay $5K, judge says". https://abajournal.com/web/article/lawyers-who-doubled-down-and-defended-chatgpts-fake-cases-must-pay-5k-judge-says
  13. Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14.02.2024. Decision: https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do (the text itself could not be opened automatically; details from a secondary source: https://en.wikipedia.org/wiki/Moffatt_v._Air_Canada)
  14. Anthropic. Reduce hallucinations (Claude documentation). https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations
  15. Lewis P., Perez E., Piktus A. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
  16. Dhuliawala S., Komeili M., Xu J. et al. Chain-of-Verification Reduces Hallucination in Large Language Models. 2023. arXiv:2309.11495 (preprint). https://arxiv.org/abs/2309.11495
  17. Manakul P., Liusie A., Gales M. J. F. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. EMNLP 2023. https://arxiv.org/abs/2303.08896
AI Sandbox Lab

Learn this by doing

In the AI Sandbox Lab practical groups you work with an agent on your own tasks instead of just reading about it. No coding required.

See the program

Читать по-русски · Чытаць па-беларуску

← All materials

Ask a question