Why AI Models Hallucinate, Really

You ask an AI coding assistant which version of a library added a specific function. It answers instantly, with total confidence, citing a changelog entry — chapter and verse. You go check. The function exists. The version number does not. There was no such release. The model didn't find that changelog entry anywhere; it built one, the way it builds everything else, and it built it to look exactly like a real one would.

That's not a bug you can patch out with a better prompt. It's the model doing exactly what it was trained to do — and understanding why is the difference between working around hallucination and being constantly surprised by it.

What Hallucination Actually Is

A large language model doesn't have a database of facts it looks things up in. It has a very good sense of what word is statistically likely to come next, given everything that came before it in the conversation. That's it. That's the whole mechanism.

Most of the time, "statistically likely" and "actually true" line up closely enough that you don't notice the difference. Ask a model to explain how a hash map works, and the likely-next-word path and the correct-technical-explanation path are basically the same road. But ask it something more obscure — an exact version number, a niche API's argument order, a paper that may or may not exist — and the model is still doing the same thing: predicting plausible-sounding tokens. It just has less real signal to anchor on, so it fills the gap with whatever pattern fits the shape of a confident answer.

Hallucination is what that filling looks like from the outside. The model isn't distinguishing between "I know this" and "I'm inferring this from vibes" — it doesn't have a knob for that. It produces the same fluent, grammatically perfect output either way. That's what makes it dangerous: a hallucinated answer doesn't come with a hedge or a stutter. It reads exactly as confident as a correct one.

The core technical pieces

Next-token prediction has no concept of "unknown"

When you ask a traditional database a question it can't answer, it returns nothing, or an error. An LLM can't do that natively, because it was never trained to output "I don't know" as a first-class outcome — it was trained to keep predicting the next most probable token, every single time, no matter what. There's no built-in circuit that fires when the model has run out of real information. It just keeps going, and going smoothly is exactly what it's optimized to do.

Training data has gaps, and the model doesn't know where they are

The model's "knowledge" is really a compressed statistical residue of the text it was trained on. If a fact appeared rarely, or in a form the model didn't generalize well from, or after its training cutoff, that fact is effectively a gap — but the gap is invisible to the model itself. It doesn't experience "I have no data here" as a distinct internal state the way you'd notice a blank page. It experiences it as slightly less certainty, spread across a probability distribution that still has to add up to 100%. Something has to win that vote, even if nothing in the distribution is actually correct.

Fluency and accuracy are trained separately, and fluency wins by default

Base language models are trained to be fluent predictors of text. The instruction-tuning and RLHF passes on top of that are trained to make responses helpful, well-formatted, and confident-sounding — because that's what human raters preferred in side-by-side comparisons. Nobody was explicitly rewarding "correctly expressed uncertainty" as often as they were rewarding "clear, complete, confident answer." So the model learned, quite reasonably given its training signal, that a clean answer beats a hedged one — even when a hedge would be the honest response.

Grounding gives the model something to point at instead of guess

This is where retrieval-augmented generation (RAG) — the thing this series started with — earns its keep. If you covered "What is RAG, Really?" already, this is the connection: RAG doesn't fix hallucination by making the model smarter. It fixes it by changing the task. Instead of "recall this fact from your training," the model's job becomes "summarize this text I just handed you." Models are dramatically better at the second task, because summarizing a passage that's sitting right there in the context window doesn't require the model to reach into a lossy, compressed memory of billions of documents. It just has to read.

A real-world example

Picture an internal support bot built for a mid-sized company's IT helpdesk. Someone asks it, "What's our policy on expensing a second monitor?" Without grounding, the model has almost certainly seen thousands of generic expense-policy documents during training — different companies, different rules, different dollar thresholds. It has no way to know none of those belong to this company. So it answers fluently, picks a plausible-sounding number like "up to $150 with manager approval," and states it as fact. It's not lying. It's doing exactly what an ungrounded model does when asked a question shaped like something it's seen many versions of.

Now give that same bot access to the actual internal policy doc via RAG. The question becomes "find the expensing section in this document and tell me what it says," and the model's job shrinks from "remember something true" to "read something true." The hallucination risk doesn't vanish — the model can still misread or misquote the retrieved text — but it drops enormously, because the model no longer has to invent the underlying fact from scratch.

Why not alternative approaches

The obvious first instinct is: just tell the model to be more careful. Add "only answer if you're sure" or "don't make things up" to the system prompt. This helps a little, at the margins, but it doesn't fix the underlying issue, because the model still has no reliable internal signal for "sure" versus "unsure" to act on. You're asking it to introspect on a distinction its training never taught it to track. It'll happily comply with the instruction while still confidently hallucinating, because from its own "point of view," it is sure — the tokens it's generating are the most probable ones available to it.

The second instinct is fine-tuning the model on a big pile of correct facts about your domain. This can genuinely help with tone and format, but it's expensive to keep current, and it still bakes facts into the same lossy statistical weights that caused the problem in the first place. Six months later, when the policy changes, you're either retraining again or shipping stale answers with total confidence — the exact failure mode you were trying to avoid, just on a slower clock.

Grounding sidesteps both problems. It doesn't ask the model to be more honest about what it knows; it reduces how much the model has to know in the first place, and it updates the moment the underlying document does, with no retraining required.

A pitfall worth knowing

Grounding isn't a hallucination-proof vest — it's a hallucination-risk reducer, and people who treat it as absolute get burned. Even with the right document sitting in the context window, a model can still misattribute a detail from one paragraph to a different one, paraphrase a number incorrectly, or blend two retrieved chunks into a sentence that sounds coherent but was never actually written anywhere in the source. This is sometimes called "closed-domain hallucination," and it's sneakier than the open-book kind, because it happens even when the correct answer was technically right there. The retrieved context lowers the odds; it doesn't remove them.

If you're building anything where a wrong answer has real consequences — financial figures, medical information, legal language — the practical fix is to have the model cite the specific source chunk it drew from, and to spot-check that the citation actually says what the model claims it says. Treat a grounded answer as "probably right, and here's where to verify it," not as "verified."

The takeaway

Hallucination isn't a glitch in an otherwise-reliable system — it's the predictable output of a model doing next-token prediction with no built-in sense of what it doesn't know. Grounding techniques like RAG don't cure that; they change the model's job from recalling to reading, which is a task it's genuinely good at. That's a real improvement, but it's a risk reduction, not a guarantee, and the gap between those two is exactly where careless AI products get burned.

Next up in this series: temperature and sampling — the dial that controls how much a model is willing to wander from the safest, most predictable answer, and why cranking it up or down changes more than just "creativity."