What generative AI actually does: the next-token prediction loop

Ask an internal documentation assistant how to configure request retries and it hands back a clean code sample: correct syntax, plausible defaults, even a link in the house style — built around a config flag that was deleted from the codebase fourteen months ago. The model never saw the removal. It saw thousands of API answers shaped like this one and learned that authoritative, docs-like text tends to follow docs-like questions. No exception, no warning log, nothing for monitoring to catch. The system worked exactly as designed. That is the problem this article addresses.

Understanding what a language model mechanically does — not at a handwavy level, but well enough to reason about failure modes — is the prerequisite for every subsequent decision you will make as an AI engineer. Before RAG, before agents, before evaluation. Start here.

Text is not what the model sees

You type a sentence. The model does not receive that sentence as characters, and it does not receive it as words. It receives a sequence of integers.

Tokenization is the step between your string and the model's input. Most current models use Byte-Pair Encoding (BPE), a compression algorithm originally designed for text: start with individual bytes, then iteratively merge the most frequent adjacent pair into a new symbol. Run this merge operation 50,000 to 200,000 times and you have a vocabulary of subword units that captures frequent patterns without requiring a dictionary of all possible words.

The practical consequences for an AI engineer:

• "tokenization" might be two tokens: ["token", "ization"]. Or one. It depends on the specific model's vocabulary and training corpus.
• Common English words are usually a single token. Rare words, code identifiers with underscores, non-Latin scripts (Thai, Japanese, Korean), and numbers fragment more heavily. A URL might be 20 tokens. A Chinese sentence costs more tokens per semantic unit than an equivalent English sentence.
• You pay per token, not per word. One English word averages roughly 1.3 tokens with GPT-family tokenizers. The ratio worsens for code and multilingual content.

{ "type": "tokenizer", "title": "How your text becomes model input" }

The famous "can't spell strawberry" failure mode comes from here, not from the model being bad at spelling. The model sees tokens; it never sees individual characters in the way humans process them. If strawberry is a single token, the model has no internal representation of the fact that it contains three r's. The /articles/tokenization-bpe-intuition article covers this fully — for now, the key insight is that every cost calculation and every context-limit decision happens in token space, not word space.

The forward pass: what the model computes

Once you have a token sequence, the model runs a single forward pass through its transformer architecture. You do not need to understand the details of attention to engineer against LLMs — but you do need to understand what comes out.

The model's final layer produces a logit vector: a real-valued score for every entry in the model's vocabulary. Vocabulary sizes range from about 32,000 (older Llama models) to 200,000+ (frontier models with multilingual and code coverage). These logits are unbounded — they can be very negative (extremely unlikely next token) or very positive (strongly predicted next token).

Softmax converts logits to probabilities:

P(token_i) = exp(logit_i) / Σ exp(logit_j)   for all j in vocabulary

After softmax, you have a probability distribution that sums to 1.0. For a well-conditioned prompt, the top few tokens might hold 60–80% of the total probability mass, with a long tail of thousands of low-probability options.

This is the raw output of the model on every single generation step. Everything else — temperature, top-p, the sampling algorithm — is a post-processing step applied to this distribution before the next token is picked.

The sampling layer: where your knobs live

Given the probability distribution, how do you choose the next token? This is where the parameters you set in your API calls actually operate.

Temperature rescales the logits before softmax:

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

At T=1.0 you get the model's native distribution. At T → 0 you get greedy decoding: the highest-logit token gets probability ≈ 1.0, everything else gets ≈ 0. The output becomes deterministic. At T > 1.0 the distribution flattens — low-probability tokens become more plausible, the model "takes more risks." Temperature does not make the model more creative in any deep sense; it changes how much probability mass flows to lower-ranked options in the existing distribution.

{ "type": "sampler", "title": "How temperature and top-p reshape the distribution" }

Top-k filters the distribution to only the k highest-probability tokens before sampling. Top-k=50 means the model can only pick from the 50 most likely next tokens, regardless of their actual probability values. Simple to implement, fragile to set: the right k for a flat distribution is different from the right k for a peaked one.

Top-p (nucleus sampling) solves that fragility by being distribution-aware. Sort tokens by probability, descending. Find the minimal set whose cumulative probability exceeds p. Sample from only that set. At top-p=0.9, if the top 10 tokens hold 90% of the probability mass, you sample from 10 tokens. If no single token dominates and you need 500 tokens to hit 90%, you sample from 500. The nucleus adapts to the distribution's shape.

Min-P (introduced at ICLR 2025) is a more recent alternative: set a minimum probability threshold as a fraction of the highest-probability token's probability. If the top token has P=0.4 and min-p=0.1, any token with P < 0.04 is excluded. This outperforms top-p at high temperatures by keeping tokens proportionally competitive rather than cutting by a fixed cumulative threshold. It has gained adoption in open-source inference stacks (llama.cpp, vLLM) and is worth knowing — though the /articles/decoding-sampling-temperature-top-p-min-p article has the full analysis.

For most production text applications: temperature 0.7, top-p 0.9, and leave top-k alone. For deterministic structured outputs: temperature 0, or use constrained decoding if your framework supports it.

The autoregressive loop

Here is the part that is easy to miss: the model generates one token at a time, in sequence. After picking token N, it appends that token to the context, runs another full forward pass, picks token N+1, and repeats. This is autoregressive generation.

sequenceDiagram
    participant SRV as Inference server
    participant DEC as Model + sampler

    SRV->>DEC: [system][user message] → forward pass
    DEC-->>SRV: token 1 ("The")
    SRV->>DEC: [system][user message]["The"] → forward pass
    DEC-->>SRV: token 2 ("refund")
    SRV->>DEC: [system][user message]["The"]["refund"] → forward pass
    DEC-->>SRV: token 3 ("policy")
    Note over SRV,DEC: Repeats until stop token or max_tokens — all inside ONE API call. Your app just receives the token stream.

This has three implications worth internalizing before you write your first production prompt:

Cost scales with output length. Every output token requires a full forward pass. Longer outputs are linearly more expensive on the compute side, and output tokens are priced 3–10× higher than input tokens by API providers — because the model had to run N separate inference steps to produce them, while input tokens are processed in a single parallel pass.

Latency scales with output length. You cannot parallelize across output tokens in the autoregressive loop. The model cannot generate token 5 until it has generated tokens 1–4. This is why reasoning models that produce long chain-of-thought traces before answering are slow: they are spending real wall-clock time on every thinking token. A 1,000-token response at 50 tokens/second is 20 seconds to completion. If you are streaming, first-token latency is fast but total-response latency is not.

The model cannot "go back." Token N is conditioned on tokens 1 through N-1. If the model makes a wrong turn early in a response, it has no native mechanism to backtrack and revise. It continues from where it is. This is why chain-of-thought prompting works — it forces the model to lay out intermediate steps before committing to an answer — and why self-consistency sampling (generate multiple independent responses, then pick the majority answer) improves factual accuracy on tasks where one generation path is error-prone.

Statelessness: the fact that changes everything

The model has no memory between API calls.

Every call you make to the API reprocesses the entire context from scratch. The system prompt, the conversation history, the retrieved documents, the current user message — all of it goes in as tokens, all of it gets attended over, every time. What looks like memory in a chat interface is an illusion maintained by the application layer: the application prepends every previous message as tokens in the new request.

This matters for three reasons:

Cost compounds with conversation length. A 10-message chat thread where each message is 200 tokens starts cheap and ends expensive. By message 10, you are paying input-token cost on ~2,000 tokens of history before the model even reads the new message. In an agentic loop with tool calls, this can escalate quickly.

Context window is the only memory. The model cannot "remember" something you told it three sessions ago unless you explicitly include it in the current context. This is why /articles/context-window-management-agents is a real engineering topic: knowing what to include, compress, and evict from the window is not a nice-to-have optimization, it is the primary mechanism for managing long-running tasks.

Retrieval matters because the window is finite. Even with 1M-token context windows available on Llama 4 Scout and Gemini 2.x/3.x, full-context inference is slow and expensive. At $3/1M input tokens, stuffing 200,000 tokens of documents into every request costs $0.60 per call. At 50,000 calls/day that is $30,000/day. RAG is not a complexity choice — it is the economically correct architecture for any application with a large document corpus. The /articles/rag-from-scratch-ingestion-retrieval-generation article picks up from here.

The distribution produces hallucination by construction

The model's job is to produce a fluent continuation of the input sequence. It does not have access to a truth oracle. It cannot "check" whether what it is about to generate is factually correct — it can only consult its learned probability distribution over what tokens tend to follow what tokens in human-written text.

Human-written text is full of confident assertions and expert-sounding explanations, complete with specific dates, names, citation formats, and policy numbers. The model learned to reproduce all of these patterns. When asked a question whose answer was not in the training data, or whose answer was ambiguous in the training data, the model will still produce a confident-sounding continuation — because confident-sounding continuations are what came after this kind of question in the training data.

This is not a bug. It is the mechanism.

Hallucination rates vary across models and task types — factual recall of well-documented entities is more reliable than niche domain knowledge — but no frontier model as of mid-2026 has solved the problem. The field describes it accurately: confident, fluent, factually wrong output is a structural property of next-token prediction on text that was written by humans who were usually right, expressed with confidence, and did not append [UNCERTAIN] to their hedges.

The design consequence: every production AI system needs a mitigation layer. Retrieval grounding (give the model source documents it must cite), output validation (check the answer against a schema or an independent verification call), human-in-the-loop for high-stakes decisions, or at minimum a UI that surfaces confidence signals and encourages verification. Treating the model as an oracle that needs a better prompt is not a production strategy.

What breaks in the prediction loop

flowchart TD
    HALL["Hallucination\nFluent output not grounded in truth"]
    LOST["Lost-in-the-middle\nAttention degrades for middle tokens"]
    REP["Repetition / degeneration\nGreedy decoding gets stuck in loops"]
    CUT["Knowledge cutoff\nTraining data ends; world moved on"]
    INJ["Prompt injection\nAdversarial tokens in context hijack behavior"]

    HALL -->|mitigation| RAG["Retrieval grounding"]
    LOST -->|mitigation| POS["Position-aware chunking\n(relevant content first/last)"]
    REP -->|mitigation| SAMP["Nucleus sampling\nrepetition penalty"]
    CUT -->|mitigation| TOOL["Tool use / RAG\nfor current information"]
    INJ -->|mitigation| GUARD["Input validation\noutput filtering"]

    style HALL fill:#dc2626,color:#fff
    style LOST fill:#dc2626,color:#fff
    style REP fill:#dc2626,color:#fff
    style CUT fill:#dc2626,color:#fff
    style INJ fill:#dc2626,color:#fff
    style RAG fill:#15803d,color:#fff
    style POS fill:#15803d,color:#fff
    style SAMP fill:#15803d,color:#fff
    style TOOL fill:#15803d,color:#fff
    style GUARD fill:#15803d,color:#fff

Lost in the middle. Research (Liu et al., 2023) established that models attending over long contexts perform worse on information placed in the middle of the input than on information placed at the beginning or end. The attention mechanism has to route across thousands of token pairs; empirically, middle-context facts get underweighted. The mitigation is not a larger context window — it is placing critical information at the prompt boundaries, and using retrieval to control which information appears and where.

Repetition and degeneration. At temperature ≈ 0, greedy decoding can get stuck: once a repetitive phrase appears, the model's prediction of the next token is influenced by that phrase, making repetition more likely, reinforcing itself. Repetition penalties (a logit discount for recently generated tokens) and nucleus sampling both help. This is rarely a problem with temperature > 0.5, but it explains why a temperature of exactly 0 on open-ended generation tasks sometimes produces oddly circular output.

Knowledge cutoff. Every model has a training cutoff date. Asking a model with a mid-2025 cutoff about events from late 2025 or 2026 gets you either a refusal, or — more dangerously — a confident confabulation. The fix is not a newer model; it is tool use or retrieval for any question that requires current information. The /articles/llm-capability-limits-and-failure-modes article covers the full taxonomy of failure modes, including context degradation and benchmark overfitting.

Prompt injection. If adversarial text appears in the model's context — in a retrieved document, in a tool result, in user-supplied content — and that text contains instruction-like tokens, the model may follow those instructions. The prediction loop does not distinguish between "instructions from the system prompt I trust" and "instructions embedded in a Wikipedia article I retrieved." This is unresolved as of mid-2026. The /articles/prompt-injection-jailbreaks article covers the attack surface.

Worked numbers: what each generation step costs

Estimating inference cost for a production request

Scenario: customer support assistant
─────────────────────────────────────────────────────────────
System prompt:           800 tokens   (persona, policy docs)
Tool definitions:        400 tokens   (3 tools, JSON schema)
Retrieved context:     2,400 tokens   (3 chunks × ~800 tokens)
Conversation history:  1,200 tokens   (6 prior turns × 200)
User message:            100 tokens
─────────────────────────────────────────────────────────────
Total input:           4,900 tokens

Response:                350 tokens

At mid-2026 illustrative pricing, GPT-4-class frontier model:
  Input:  4,900 × $3.00/1M  = $0.0147
  Output:   350 × $15.00/1M = $0.0053
  Total per call:           ≈ $0.020

At 5,000 daily active users × 8 messages/session:
  40,000 calls/day × $0.020 = $800/day → $24,000/month

Optimization levers:
  Prompt caching on the stable prefix (system + tools = 1,200 tokens),
  at a typical ~90% discount on cached reads:
    → saves 1,200 × $3.00/1M × 0.9 ≈ $0.0032/call (~22% of input cost)
    → $800/day drops to ~$670/day
    (Caching history too saves more — but only if the prompt is ordered
    stable-first: system → tools → history → volatile retrieved context)
  Switching to a mid-tier model for simple queries (routing):
    → Additional 40-60% reduction on routed traffic
  Output length cap at 300 tokens:
    → Saves ~$0.001/call, ~$1,200/month at full volume

This is why token economics matter before you pick a model.

The numbers move fast — prices fell 30–60% across all tiers from 2024 to mid-2026 as competition between providers intensified. The ratios are more durable than the absolutes: output is 3–10× more expensive per token than input, and caching stable prefixes is the single highest-yield cost reduction in most production architectures.

What this means before you write a line of code

The prediction loop has a clean set of implications for AI engineering practice.

Prompt design is distribution engineering. When you write a system prompt, you are not giving the model instructions it reads and interprets. You are providing tokens that shift the conditional distribution: given these tokens, what tokens tend to follow? Good prompts move the distribution toward the target output space. Instructions that would be unambiguous to a human might not shift the distribution the way you intend — because the model has no concept of "intent," only of token sequence co-occurrence in training data.

Output length is a cost dial, not a safety dial. Capping max_tokens does not prevent the model from making errors — it just truncates whatever the model would have produced. Setting max_tokens=100 on a task that needs 200 tokens to answer correctly gets you a truncated wrong answer, not a more careful short answer.

Temperature 0 is not the same as "accurate." Greedy decoding picks the most probable token at each step, but "most probable" is not the same as "correct." If the training distribution has a confident-sounding wrong answer as the highest-probability completion for your prompt, temperature 0 will give you that wrong answer with perfect consistency. Reproducibility is not factual accuracy.

The model you are calling has already been post-trained. The raw next-token prediction model is not what you access via any production API. Frontier models go through instruction fine-tuning (to make them follow instructions), RLHF or DPO (to align them with human preferences), and safety training. These post-training steps shift the base distribution significantly — the model you call via the API is already shaped to be helpful, to refuse certain requests, to follow formatting conventions. The /articles/ai-engineer-role-vs-ml-engineer article covers where the AI engineer's work starts relative to that pre-existing pipeline.

The loop — tokenize, forward pass, sample, append, repeat — is simple enough to hold in your head and stable enough to plan against. Everything you build on top of it — context strategies, retrieval architectures, evaluation frameworks, cost controls — is an engineering response to its specific properties. Know the loop, and the rest follows.

The models built on this loop

All major frontier models — GPT-5.x, Claude Opus/Sonnet 4.x, Gemini 3.1 Pro, Llama 4, DeepSeek V3, Mistral Large 3 — are variants of the same transformer architecture running the same autoregressive prediction loop. What differs is the scale of training, the post-training recipe, the context window, and the modalities they encode alongside text.

Reasoning models (OpenAI o3, DeepSeek-R1, Claude Opus 4 extended thinking) extend the loop with a different post-training strategy: they are trained to spend generation budget on chain-of-thought tokens before producing a final answer. The loop itself is identical — they just generate many more intermediate tokens, each one influencing the next. DeepSeek-R1 (January 2025) was the proof point that reasoning capability can emerge from reinforcement learning alone, without supervised examples of correct reasoning steps, at a fraction of the compute cost of prior approaches. The mechanism is still the prediction loop; the training signal changed.

The /articles/model-landscape-2025-2026 article covers where each model sits competitively, with real benchmark numbers and cost comparisons. The key point for this article is simpler: the mental model you need to evaluate any of them is the same one. They all tokenize input, produce logit distributions, sample tokens, and repeat. The engineering constraints — context limits, output latency scaling linearly with token count, statelessness, hallucination as a baseline property — are not provider choices. They are properties of the mechanism.

The judgment call: what to internalize vs. what to look up

You will not remember BPE merge rules. You will not remember the exact softmax formula. That is fine.

What you should internalize, because you will reason from it daily:

The model generates tokens probabilistically, one at a time, by sampling from a distribution over its vocabulary. Temperature and top-p reshape that distribution. The context window is the model's only state. Output tokens are expensive and slow. The model cannot verify what it generates. These five facts explain the most common production failures and the most important design decisions.

Everything else — which model, which sampling parameters, whether to use a reasoning model or a standard one, what retrieval strategy to pick — is application of those five facts to your specific constraints. The /articles/open-vs-closed-models-decision-framework article provides the framework for the model-selection decision once you know what you are optimizing for. The /crash-course/01-prediction-loop-genai-foundations module walks through the same loop interactively if you want hands-on reinforcement before moving to the next section.

The documentation assistant from the opening recommended a deleted config flag because the model's prediction of "what docs-like text follows a docs question" was not grounded in a source of truth — it was grounded in the training distribution, which included authoritative-sounding text that was no longer accurate. The fix is not a better prompt. The fix is retrieval that fetches the current documentation and provides it as ground-truth context. That fix requires understanding the loop well enough to know why the model was wrong in the first place.