Why do AI chatbots make things up?
They are built to produce text that fits, not text that is true, and a fluent invention fits just as well as a fact.
Simple intuition
The plain reason, in everyday words
A language model is a machine for continuing text. Shown the start of a sentence, it produces the words that most plausibly come next, based on the enormous quantity of writing it was trained on. That is the whole job. It has no separate list of facts to consult and no step where it checks a claim before saying it. When you ask for something it has seen stated clearly and often, the most plausible continuation is the true one, and you get a correct answer. When you ask for something obscure — a specific citation, a small company's founding date, a court case — there is no strongly learned answer to reach for, but the model still has to produce something, and what it produces is a sentence shaped exactly like a correct answer. Fluency is the default output, and truth is only usually a side effect of it.
The model is lying, or knows the truth and chooses not to say it.
Lying requires knowing the truth and intending to conceal it. There is no separate representation of the fact being contradicted — the fabrication is produced by the same process, in the same way, as the correct answers.
It happens because the training data contained the false claim.
Most fabrications are novel: the specific fake citation appears nowhere in the training data. It is assembled from the pattern of real ones, which is why it looks so convincing and why removing bad data does not fix it.
Asking the model whether it is sure will catch the error.
The follow-up answer is generated by the same process as the original, and it is influenced by the conversation so far. A model that just produced a fake citation will often confirm it, as happened in the New York case.
A bigger model will not do this.
Scale reduces the rate substantially on common knowledge but does not remove it, and it can make the remaining errors harder to spot by making everything sound more authoritative. Grounding in retrieved sources and tool use help more than size alone.
It sets the correct expectation for using these systems: they are excellent at producing well-formed language and unreliable at producing specific verifiable facts, so any specific — a number, a citation, a name, a date — is a claim to check rather than a result to trust. Knowing why the failure happens also tells you where it is most likely: the more obscure the fact and the stronger the surface pattern around it, the more convincing the fabrication will be.
Who worked it out
The word 'hallucination' entered this field through image captioning and machine translation in the 2010s, where systems were observed describing objects absent from the picture, and it was carried over to text generation.
What problem forced it
The problem became urgent rather than academic when large models moved from research demonstrations into products used for research, drafting and advice, where an invented citation has consequences.
How it changed since
Responses have moved outward from the model: first better training data, then reinforcement learning from human feedback, then grounding in retrieved documents and external tools, and most recently a serious argument that benchmark scoring itself rewards guessing and needs to change before models will learn to say they do not know.
Retrieval-augmented generation
The main practical defence: give the model real documents to summarise instead of asking it to recall.
How models represent uncertainty
Models carry internal signals about their own confidence; the open problem is connecting those to what they actually say.
Written for Curio rather than collected from a forum — it is part of the curated corpus that ships with the platform. The references it draws on are listed under Sources.