Why Models Hallucinate: A Mechanistic View

A developer at a fintech company deployed a chatbot with an internal policy document as a RAG source. The bot answered every question correctly in staging. Six weeks after launch, a compliance officer noticed the bot was confidently citing a liability clause that did not appear anywhere in any company document — not in the retrieved context, not in training data. The clause it invented was plausible. It sounded exactly like the other clauses around it. Nobody had asked for it; the model just completed the pattern. Nothing had crashed. There was no error log.

That is what hallucination looks like in production. Not a failure with a stack trace — a confident wrong answer that passes all the checks your system was designed to run.

What the training objective actually optimizes

A transformer is trained by next-token prediction: given a sequence of tokens, predict the next one. The loss function penalizes the model when the token it assigns highest probability to is not the correct one. Over enough data and iterations, the weights learn a compressed representation of the training distribution.

This is the first mechanistic root of hallucination. The model did not learn facts. It learned correlations between tokens that appeared in text. A fact that appears in ten million documents gets strong gradient signal; a fact that appears in two papers gets weak signal or noise. When you query the model about a niche topic, you are asking it to extrapolate from a very thin region of its training distribution. The most probable completion of "The lead author of that 2019 ICLR paper on sparse attention was" is whatever name pattern correlates most strongly with that context — which may have nothing to do with the actual author.

The training objective does not include a penalty for plausible-but-wrong tokens. A generated sentence that sounds accurate but contains an invented date loses no more than a generated sentence that is grammatically awkward. The gradient signal only knows "did you predict the training document's next token." It does not know "is what you generated factually correct in the real world."

sequenceDiagram
    participant T as "Training signal"
    participant M as "Model"
    participant O as "Output distribution"

    Note over T,M: Training: next-token prediction only
    T->>M: "correct token = 'Paris'"
    M->>M: update weights toward P(Paris | ...) ↑
    Note over M: No signal: is 'Paris' the capital of France?
    Note over M: No signal: is 'Lyon' plausible but wrong?
    Note over M,O: At inference: model outputs highest P(token | context)
    M->>O: P("Paris")=0.91, P("Lyon")=0.06, P("Bordeaux")=0.03
    Note over O: For rare facts: P("wrong")=0.55, P("right")=0.45

Probabilistic decoding never picks the truth — it picks from a distribution

Even when the model's internal representation has encoded the right answer, the decoding step is a draw from a probability distribution, not a lookup. The decoding parameters — temperature, top-p, min-p all manipulate that distribution before the draw.

Temperature flattens (T > 1) or sharpens (T < 1) the distribution. At T = 1, you sample from the raw softmax output. At T = 0, you take the greedy argmax. The widespread belief is that temperature=0 eliminates hallucination. It does not. Temperature=0 eliminates sampling variance. It commits you to the highest-probability token at every step. If the highest-probability token is wrong — which happens whenever the correct token is not the argmax, something only guaranteed impossible when the model puts more than 50% of its mass on the right answer — then you get the wrong answer every single time. At T=0 you have made the model maximally committed to whatever its distribution says, right or wrong.

What temperature actually controls is the hallucination rate for borderline facts, and the confidence of hallucinated output. At T > 1, the model hallucinates differently per run. At T = 0, it hallucinates the same wrong answer consistently and with what looks like confidence. Neither is safe for production factual queries.

Worked example — rare fact, model has weak signal:

  Most plausible wrong answer:    0.42  (top-1, wins greedy)
  Correct answer probability:     0.38
  Other plausible wrong answers:  0.20 (distributed)

At T=0 (greedy):   Wrong answer selected 100% of runs (0.42 > 0.38)
At T=1 (sampling): Correct answer selected 38% of runs, wrong 62%
At T=0.5:          Distribution sharpened → wrong answer probability rises further
                   Wrong: ~0.52, Correct: ~0.44 → still picks wrong at greedy

The model does not know more at low temperature.
It just commits more fully to what it does know.

The missing verification loop

When you retrieve a fact from a database, there is a mechanism — the database index — that maps a query to the stored record. If the record exists, you get it. If it does not, you get a null. The database does not make something up when a lookup fails.

A transformer has no equivalent mechanism. There is no flag in the residual stream that says "I am generating from verified knowledge" versus "I am pattern-completing from weak signal." The model cannot introspect its own confidence in a way that stops it from generating. It generates until the sequence ends or a stop token appears.

Mechanistic interpretability research has produced one of the more unsettling findings in this space: models can encode the correct answer in the residual stream and still output the wrong token. The residual stream, as covered in the transformer block internals, is a shared vector that accumulates updates from each attention and MLP sublayer. A fact can be present as a feature in that vector at one layer and then overwritten or suppressed by a subsequent layer's update before the final logit projection reads it.

Think of the residual stream as a whiteboard where 32 or 48 layers are writing and erasing simultaneously. The final projection — the unembedding matrix that converts the residual stream vector to logits over the vocabulary — reads whatever is on the whiteboard at the last layer. If an early-layer attention head wrote "Grant" (the answer to "Who is buried in Grant's Tomb?") but a later MLP layer overwrote part of that feature space with a more recently seen name, the unembedding reads the wrong thing.

Attention sinks: when attention goes nowhere useful

One well-documented quirk of attention allocation is the attention sink phenomenon, identified and named in 2023 by Xiao et al. in the StreamingLLM work (with BERT-era antecedents — heads dumping attention on [SEP] and [CLS]). Specific tokens — most commonly the BOS (beginning-of-sequence) token, certain punctuation, and sometimes the first real content token — absorb a disproportionate fraction of total attention mass from many heads across deep layers. The accepted reading is that sinks act as a functional resting place: softmax forces every head to put its attention somewhere, and a semantically empty token is the least destructive place to park it. Remove the sink tokens and generation quality collapses — which is why StreamingLLM keeps them pinned in the cache.

Why bring sinks into a hallucination article? Because attention is a budget. Every head's attention scores must sum to 1 across all positions. When a sink token absorbs 30–40% of the attention mass of many heads, the remaining 60–70% must cover the entire context. For a 10,000-token context, that concentration effect means the model is spreading thin attention over large swaths of the input.

flowchart LR
    BOS["BOS token\n(attention sink)"] 
    T1[Token 1]
    T2[Token 2]
    FACT["Critical fact\nat position 6,000"]
    T3[Token 3]
    
    HEAD["Attention head\n(Layer 24)"]
    HEAD -->|"35% of mass"| BOS
    HEAD -->|"18% of mass"| T1
    HEAD -->|"22% of mass"| T2
    HEAD -->|"4% of mass"| FACT
    HEAD -->|"21% remaining"| T3
    
    style BOS fill:#ff2e88,color:#fff
    style FACT fill:#ffaa00,color:#0a0a0f
    style HEAD fill:#a855f7,color:#fff

Be careful about the causal story here. What sinks demonstrate is that attention mass is allocated very unevenly — a fact physically present in the context can end up with a tiny attention weight, and its contribution to the residual stream at that layer is correspondingly small. Whether sinks cause the model to neglect in-context facts is an open question; the evidence supports "sinks are a symptom of how softmax attention distributes its budget," not "sinks steal attention that would otherwise have found your fact."

The context-neglect phenomenon itself has its own literature: "lost in the middle" (Liu et al., 2023) showed that information placed in the middle of long contexts reliably gets lower retrieval rates than information at the beginning or end. The context window article covers the KV cache implications; uneven attention allocation is one reason the problem exists at the representation level before you even get to retrieval.

Temperature and the creativity-factuality tradeoff

There is no dial labeled "creativity" on a language model. What most people call creativity is high temperature plus a diverse training corpus. What people call factuality is low temperature plus a narrow, well-grounded prompt. These trade off because they both operate on the same underlying mechanism: the shape of the probability distribution over the vocabulary.

A distribution can be sharp (one token has most of the probability mass) or flat (probability spread across many tokens). Sharp distributions generate predictable, conservative text. Flat distributions generate varied, creative text — and also more hallucination, because the probability of a low-confidence wrong token becomes non-trivial.

This is why you cannot simultaneously maximize creative output and minimize factual error. Any mechanism that makes the distribution flatter (higher temperature, diverse in-context examples, open-ended prompts) also raises the chance that a wrong token gets sampled. The sampling parameter tradeoffs matter here: min-p, now the preferred sampling approach for creative tasks in open-source deployments, adapts the cutoff to the distribution shape rather than applying a fixed nucleus threshold. It is better at cutting off low-probability garbage without collapsing to greedy. But it does not change the fundamental tradeoff — it just manages the distribution more carefully.

{ "type": "sampler", "prompt": "story-open", "temperature": 1.0, "title": "How temperature reshapes the token distribution" }

What RAG actually fixes (and what it does not)

Retrieval-Augmented Generation addresses hallucination by a different mechanism than tuning decoding parameters. Instead of modifying the shape of the probability distribution, RAG changes what information is available in the context. If the retrieved passage says "the interest rate was 4.5% in Q3 2025," then the model's distribution shifts dramatically toward tokens that continue that fact accurately.

The reduction in factual confabulation is real. For tasks with a clear information source, well-implemented RAG brings error rates from several percent down to fractions of a percent for in-passage facts. The RAG pipeline is the right architecture when factual grounding from a controlled corpus matters.

But RAG introduces its own failure mode: attribution hallucination. The model, presented with retrieved text, can confabulate details that were not in the retrieved passage while appearing to draw on it. It might cite the retrieved document as the source of a claim that was actually generated from the model's parametric memory. It might blend two retrieved passages and attribute the blend to one of them. It might add a plausible-sounding detail to an otherwise accurate retrieved fact.

Attribution hallucination is harder to detect than confabulation because the model's output looks grounded — it is citing something real. Evaluation systems that check "does the answer match the retrieved context?" using string matching will miss it. You need semantic entailment checking: does the generated claim logically follow from the retrieved text, or did the model add something beyond what was retrieved?

flowchart TD
    Q[User query] --> R[Retrieve relevant chunks]
    R --> C{Model generates answer}
    C -->|"Content from retrieved text"| OK["Grounded claim\n(correct)"]
    C -->|"Content model invented\nbut sounds retrieved"| BAD["Attribution hallucination\n(hard to detect)"]
    C -->|"Content outside retrieved text\nfrom parametric memory"| MAYBE["Parametric claim\n(may or may not be correct)"]
    
    style BAD fill:#ff2e88,color:#fff
    style OK fill:#15803d,color:#fff
    style MAYBE fill:#ffaa00,color:#0a0a0f

Calibration: the overlooked production problem

Accuracy and calibration are different things. A model is accurate if it gives the right answer. A model is calibrated if its expressed confidence (reflected in output logprobs) tracks its actual error rate. A perfectly calibrated model that says it is 70% confident is correct 70% of the time. An overconfident model that says it is 95% confident might be correct only 60% of the time.

Current production LLMs are systematically overconfident, and the mechanism is largely post-training. Pretrained base models are surprisingly well calibrated — their logprobs track empirical accuracy decently, because next-token prediction on a huge corpus is itself a calibration exercise. RLHF then breaks this: the GPT-4 technical report showed the base model's near-diagonal calibration curve visibly degrading after RLHF, with the post-trained model pushing probability mass toward confident answers. The incentive structure compounds it. As the 2025 OpenAI argument in "Why Language Models Hallucinate" lays out, benchmarks graded binary-right-or-wrong award zero points for "I don't know" and partial credit in expectation for a confident guess — so post-training toward those evals teaches the model that guessing dominates abstaining. Call it a fifth mechanistic root, alongside the four above: the pretraining objective creates the hallucinations, and post-training teaches the model to deliver them with a straight face.

The expressed confidence — recoverable from the output logprobs — therefore does not reliably track actual correctness. This matters for any production system that uses confidence as a routing signal: "if the model is high-confidence, return the answer; if low-confidence, escalate to human review." With a miscalibrated model, that logic fails to catch exactly the cases where the model is confidently wrong — which are usually the most dangerous.

The practical implication: do not rely on model self-reported confidence without calibration data. If you need a reliability signal, measure it empirically on your domain. Run a set of queries where you know the ground truth, plot model confidence against empirical accuracy, and build a calibration curve. Then apply that curve to live traffic. This is standard in ML systems and almost never done for LLM deployments.

Calibration check (worked example):
  Sample 500 queries where ground truth is known.
  For each, get model's top-token logprob as confidence score.
  Bucket into confidence ranges: [0.9-1.0], [0.8-0.9], ... [0.0-0.1]
  Measure empirical accuracy per bucket.

  Ideal (calibrated):
    Confidence 0.90-1.00 → accuracy ~92-95%
    Confidence 0.70-0.80 → accuracy ~73-77%

  Typical LLM (overconfident):
    Confidence 0.90-1.00 → accuracy ~68-72%  ← the dangerous zone
    Confidence 0.70-0.80 → accuracy ~54-60%

  Fix: learn a monotonic remapping from raw confidence to calibrated confidence.
  Then use calibrated score as routing threshold.

What breaks

There are several failure patterns specific to hallucination in production, each with a different root cause and a different fix.

Knowledge cutoff confabulation. The model's training data ends at a date. Queries about events after that date produce confident-sounding answers that are entirely invented. The model does not know what it does not know. Fix: date-aware routing — if the query is about a recent event, force retrieval before generation.

Entity interpolation. The model knows two real entities — say, two researchers in the same field — and confabulates a paper they supposedly co-authored by combining their names and real paper titles from each. The result looks real but does not exist. This is especially common for niche academic or professional queries. Fix: entity verification against a structured source before surface to user.

Citation fabrication. When asked for a reference, the model generates plausible-looking titles, author names, and venue names by pattern-completing from its training distribution on academic text. The citation follows the right format and often names real people who do work in the area. Fix: never accept model-generated citations without verification against a real index.

Instruction drift under long context. As context grows, attention sinks and mid-context degradation cause the model to gradually lose track of constraints specified in the system prompt. A constraint like "only answer based on the provided document" may be followed at token 500 and violated at token 8,000. Fix: repeat critical constraints near the end of the context window, or use structured output formats that enforce the constraint mechanically. See context window management for agents for more on this.

Hallucination in tool outputs. A model using tools can hallucinate the content of a tool call's return value if it fails to actually parse the result and instead predicts what the result "should" say. This is a subtle failure mode in agent frameworks where the model has seen many similar tool responses in training and pattern-completes rather than reading the actual current response. Fix: structured output constraints on tool response parsing, explicit pointer tokens that force the model to ground in the literal response string.

The decision in practice

Hallucination is not one problem with one fix. The right mitigation depends on which mechanistic root is dominant for your use case.

If you need factual grounding on a known corpus, RAG is the right tool — but measure attribution hallucination specifically, not just answer correctness, using semantic entailment evaluation rather than string matching. Multi-step reasoning calls for a different lever: chain-of-thought externalization surfaces intermediate errors before they compound, and it should be paired with a verifier — a separate model pass or a structured output that checks internal consistency — rather than trusted on the final answer alone.

High-stakes decisions where wrong-and-confident is the worst outcome make calibration measurement non-optional. Know your model's actual accuracy per confidence bucket on your domain before deploying a confidence-based routing system. At the other extreme, creative tasks can tolerate confabulation: keep temperature higher, accept that occasional factual error is the cost of varied, non-repetitive output, and scope the use case so users and downstream systems know they are getting generated text, not verified claims.

The hardest case is queries about rare entities, niche topics, or recent events — exactly where parametric recall is thinnest. Treat the model's output there as a draft that needs verification, not a final answer, and reach for tool use: a web search, a database query, a structured API call. The tool and function calling article covers the mechanics; the strategic point is that tool use is not just for "doing things" — it is for grounding factual claims in live, verifiable sources.

One pattern that works well in production: the two-pass architecture. The model generates an answer in the first pass. A second, separate verification pass checks whether each factual claim in the answer is supported by the context. Claims without support get flagged or stripped. This is more expensive than single-pass generation — add roughly 30-50% to your token count — but it converts an invisible hallucination problem into a measurable, controllable one. The output-validation patterns at scale are covered in detail in the hallucination mitigation article.

The field has not solved hallucination. A line of work grounded in computability theory and calibration — Kalai & Vempala's "Calibrated Language Models Must Hallucinate" (2023) and Xu et al.'s "Hallucination is Inevitable" (2024) — argues it cannot be fully solved for any model trained on next-token prediction: the training signal does not distinguish plausible-but-wrong from plausible-and-right. What has improved is the tooling to detect, measure, and contain it. Engineers who understand the mechanism — incomplete data, probabilistic decoding, no verification loop, attention allocation failures, post-training incentives that reward the confident guess — build systems that account for it rather than systems that assume it will not happen.