The Prediction Loop: What Generative AI Actually Does

Ask a state-of-the-art language model to count the letter R in "strawberry" and there is a reasonable chance it gets it wrong. Three R's — one in "straw," two in "berry" — and yet a model that can write production Python, explain quantum entanglement, and pass the bar exam stumbles on this. This is not a random quirk. By the end of this module you will know exactly why it happens, and that explanation will give you the mental model for every design decision you will ever make as an AI engineer.

The one-sentence model

Generative AI — every model you will ever call, from GPT-5 to Claude to Llama 4 to the smallest open-weight model running on your laptop — does exactly one thing:

Predict the next token. Repeat.

That's it. The sophistication is in what "predict" means at scale, but the loop itself is embarrassingly simple:

flowchart LR
    A["Tokens so far<br/>(context)"] --> B["Model forward pass<br/>(billions of params)"]
    B --> C["Probability distribution<br/>over ~50K–150K tokens"]
    C --> D["Sample one token"]
    D --> E["Append to context"]
    E --> A
    style B fill:#0e7490,stroke:#0e7490,color:#fff
    style C fill:#0e7490,stroke:#0e7490,color:#fff
    style D fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f

Each iteration of that loop produces one token. A typical response runs 300–800 tokens, so you're looking at 300–800 forward passes through the model per reply. The latency you experience in a chatbot is mostly the time spent running this loop, one token at a time. The cost you see on your API bill is directly proportional to how many tokens — input plus output — went through that loop.

What a token actually is

Before the loop can run, text has to become numbers. The model does not see characters. It does not see words. It sees tokens — chunks produced by an algorithm called Byte Pair Encoding (BPE), which starts from raw bytes and greedily merges the most frequent adjacent pairs until it hits a vocabulary limit, typically 50,000–150,000 entries.

The vocabulary is a learned artifact of the training data distribution, with no linguistic design behind it. Common English words become single tokens because they appear together constantly. Rare words, proper nouns, and strings that appeared infrequently in training get split into pieces.

This is why "strawberry" breaks letter-counting. The model sees something like:

"strawberry" → ["st", "raw", "berry"]   (3 tokens, GPT-4o tokenizer)

Mid-sentence it gets worse: " strawberry" with its leading space is a single token on the GPT-4o tokenizer. There is no token-level representation of individual characters. To count R's, the model would need to look inside tokens — below the level of its own input representation. It was never directly trained to do this. It can sometimes get it right by reasoning through the character composition from what it learned about common words during training, but it is guessing from pattern memory, not actually examining characters. The failure comes from a mismatch between the question (character level) and the representation (subword token level).

{ "type": "tokenizer", "title": "Why models can't count R's: text becomes tokens, not characters", "text": "strawberry" }

The tokenizer matters for practical reasons beyond spelling games. English averages roughly 1.3 tokens per word (about 0.75 words per token) on GPT-style tokenizers. Arabic, Thai, and Korean can hit 2–3× that — a hidden cost and quality tax on non-English use cases that can make your cost estimates wildly wrong if you do them in English-equivalent terms. Numbers also fragment: 1,234,567 often becomes several tokens, which is why arithmetic and "how many digits" tasks are harder than they look.

The distribution view

After tokenization, the forward pass through the model produces a logit vector — one score per vocabulary entry, across all 50K–150K possible next tokens. A softmax converts those scores to probabilities. The model then samples one token from that distribution.

This framing matters because every failure mode you will encounter traces back to something in this distribution:

• Hallucination: the training distribution included plausible-but-false text. The model predicts tokens that complete a fluent pattern, regardless of whether those tokens describe reality. There is no internal verification step — the model cannot distinguish "I know this" from "this completes the pattern."
• Prompt sensitivity: a small wording change shifts the probability mass in the logit vector, potentially changing which token gets sampled.
• Temperature bugs: setting temperature=0 makes the model always pick the argmax (greedy decoding), which is deterministic but brittle. Setting temperature=2.0 flattens the distribution so aggressively that you get incoherent token soup.

The model has no beliefs in the human sense. It has a learned probability distribution over text sequences, conditioned on everything in the context window. That is genuinely powerful — but it is not a knowledge store you can query reliably, and no amount of prompting converts it into one.

Sampling: what temperature, top-p, and min-p actually do

Every sampling parameter is a geometric transformation applied to the probability distribution before drawing a token — nothing more mystical than that.

Temperature scales the logits before the softmax. Below 1.0 sharpens the distribution toward the most likely tokens. Above 1.0 flattens it, spreading probability across more of the vocabulary.

Top-K truncates to the K highest-probability tokens before sampling. Prevents the model from drawing from the long tail of improbable tokens, but the cutoff is fixed regardless of whether the distribution is narrow or wide.

Top-P (nucleus sampling) keeps the smallest set of tokens whose cumulative probability reaches P. More adaptive than top-K because the number of candidates shrinks when the model is confident and grows when it is uncertain.

Min-P (from a 2024 paper, now widely adopted in open-source runtimes like llama.cpp and vLLM) sets a minimum probability threshold relative to the top token. If the most likely token has probability 0.8 and min-p is 0.1, any token below 0.08 gets cut. This adapts dynamically to distribution shape in a way top-P cannot, and it produces fewer nonsensical outputs at high temperatures.

{
  "type": "sampler",
  "title": "How temperature and sampling reshape the token distribution",
  "prompt": "capital-france",
  "temperature": 0.7,
  "topP": 0.9
}

Practical guidance: for factual extraction or structured output, use temperature=0.0–0.2. For code generation, 0.0–0.4. For creative tasks, 0.7–1.0. For reasoning models (o3, DeepSeek-R1, Claude Opus with extended thinking), do not set temperature at all — providers internally constrain the sampling process and exposing a temperature parameter to the caller is counterproductive.

The loop in code

Here is the actual generation loop, simplified to what matters conceptually:

from anthropic import Anthropic

client = Anthropic()

# The context is everything — history, system prompt, new message
messages = [{"role": "user", "content": "Explain attention in one sentence."}]

# One API call runs the prediction loop until a stop condition
response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=256,
    messages=messages,
)

# response.content[0].text is the concatenation of all sampled tokens
print(response.content[0].text)
print(f"Input tokens: {response.usage.input_tokens}")   # what you paid to process
print(f"Output tokens: {response.usage.output_tokens}")  # what the loop produced

The key insight in the numbers: you pay for every token in the context window on every call, not just the new message. If your conversation history is 3,000 tokens and you send one more message, you pay for 3,000 + 1 + system prompt + new output. Context costs accumulate. This is why long-agentic workflows can get expensive fast, and it is the foundational economic reason that prompt caching and context management exist as disciplines.

Why "autocomplete on steroids" is both right and wrong

The autocomplete framing is technically accurate and practically misleading at the same time.

It is accurate because the mechanism is genuinely the same as your phone's keyboard prediction, scaled enormously. GPT-4 and your phone's next-word suggestion are both predicting the next token in a sequence. One is a small model trained on a user's personal message history; the other has hundreds of billions of parameters trained on most of the text the internet has ever produced.

But the behaviors that emerge at that scale are not just "better autocomplete." When you train next-token prediction on enough data with enough parameters, the model develops internal representations that can:

• Translate between languages it has never been explicitly taught to translate
• Write and debug code across dozens of programming languages
• Solve logic puzzles by working through them step by step
• Summarize and compare ideas across domains it has never seen paired

None of this was explicitly programmed. It emerged from the prediction task at scale. The loop is simple; what the loop can express is not.

The reason the framing is misleading: it implies the model is just guessing likely continuations with no real understanding. But "likely continuation" conditioned on a training distribution of human knowledge turns out to encode a lot of structure about the world. The model has something closer to a compressed, probabilistic world model than a Markov chain over word pairs. Not understanding in the human sense — but not nothing.

The honest position: it is a very sophisticated learned distribution over text. That is more powerful than "autocomplete" suggests, and more brittle than "understanding" implies.

One caveat before you build a mental model on pretraining alone. Raw next-token pretraining produces a base model, and a base model does not answer questions — it continues text. Prompt one with "What is the capital of France?" and a likely continuation is another quiz question, because that is what such text looks like on the internet. The assistant you actually talk to went through post-training: supervised fine-tuning on instruction–response pairs, then preference optimization (RLHF and its cheaper descendants), which reshape the distribution so that "a helpful answer" becomes the most probable continuation of "a question." Refusals, the assistant persona, the compulsive bullet lists — all post-training artifacts. The prediction loop itself never changes; what changes is which continuations are likely. RLHF and reward models covers how that pass works.

What breaks in practice

flowchart TD
    LOOP["Next-token prediction loop"] --> H["Hallucination"]
    LOOP --> PS["Prompt sensitivity"]
    LOOP --> CTX["Context blindness"]
    LOOP --> CUT["Knowledge cutoff"]
    H --> H1["Plausible-sounding wrong facts"]
    PS --> PS1["Tiny wording changes, big output swings"]
    CTX --> CTX1["Lost-in-the-middle effect past ~32K tokens"]
    CUT --> CUT1["No knowledge of events after training"]
    style LOOP fill:#0e7490,color:#fff
    style H fill:#ff2e88,color:#111
    style PS fill:#ff2e88,color:#111
    style CTX fill:#ff2e88,color:#111
    style CUT fill:#ff2e88,color:#111

Hallucination is not a bug to patch out — it is a structural consequence of probabilistic next-token prediction over incomplete training data. No frontier model as of mid-2026 has solved it. A model with correct internal representations can still hallucinate because sampling draws a lower-probability wrong token. The system-level mitigations are retrieval-grounded generation (RAG), self-consistency checks across multiple samples, and human-in-the-loop verification for high-stakes outputs. See why models hallucinate for the mechanistic view.

Context blindness in the middle: models advertise context windows of 128K, 1M, even 2M tokens. In practice, recall degrades badly for information placed in the middle of a long context — the "lost in the middle" effect is reproducible across all major models. Needle-in-haystack benchmarks show many 128K models losing reliable retrieval past 32–64K. The headline number is not a quality guarantee; it is a capability boundary, and effective recall depends on where in the window the relevant information sits.

Prompt sensitivity: the model sees every character in the prompt. Changing "write me Python code" to "write Python code" can produce different output. Adding a period can change behavior. Careful context engineering turns this from a source of flakiness into something you can work with deliberately. But it does mean you cannot evaluate a prompt once and call it done.

Knowledge cutoff: every model has a training cutoff date. Production systems requiring current information — prices, recent events, updated documentation — need retrieval augmentation or tool use, not just a larger context window.

The AI Engineer role: what it is and what it is not

Before 2023, the term barely existed. By 2026, AI Engineer is a distinct, well-compensated role ($140K–$220K in the US market) with a specific toolbox. The distinction that matters:

• ML Engineers train and deploy custom models. They own data pipelines, feature engineering, training loops, evaluation, and retraining schedules. They interact with model weights directly.
• AI Engineers build products on top of pre-trained foundation models via APIs. The work is: prompt design, RAG pipelines, vector databases, agent orchestration, evaluation harnesses, and production reliability.
• AI Researchers design new architectures, training methods, and alignment techniques. They need deep math and they publish.

Most of what you will do in this course is AI Engineering. You will call APIs. You will never see a gradient. The hard parts are system design, evaluation, and production reliability — not model training.

flowchart LR
    subgraph "AI Researcher"
        R1["New architectures"]
        R2["Training methods"]
        R3["Alignment research"]
    end
    subgraph "ML Engineer"
        ML1["Data pipelines"]
        ML2["Training loops"]
        ML3["Model evaluation"]
        ML4["Custom deployments"]
    end
    subgraph "AI Engineer"
        AI1["Prompt & context design"]
        AI2["RAG pipelines"]
        AI3["Agent orchestration"]
        AI4["Evals & LLMOps"]
        AI5["Production reliability"]
    end
    style AI1 fill:#0e7490,color:#fff
    style AI2 fill:#0e7490,color:#fff
    style AI3 fill:#0e7490,color:#fff
    style AI4 fill:#0e7490,color:#fff
    style AI5 fill:#0e7490,color:#fff

The practical skill delta for AI Engineers in 2026: RAG architecture, vector databases (Qdrant, Pinecone, Weaviate), evaluation harnesses (Ragas, Braintrust), structured outputs, prompt caching, agent loops, Model Context Protocol (MCP) for tool integration, and production reliability patterns. Most of this toolbox did not exist as a job requirement before 2023. It is moving fast. The underlying mechanism — the prediction loop — is stable. The tooling on top of it is not.

The model families in one honest paragraph

By mid-2026 there are three tiers. Closed frontier (GPT-5.x, Claude Sonnet/Opus 4.x, Gemini 3.1 Pro): best on complex reasoning and coding benchmarks, priced around $3–15 per million output tokens (illustrative, as of mid-2026), improving monthly. Open-weight (Llama 4 Scout/Maverick, DeepSeek V4, Mistral Large 3, Qwen 2.5/3): the performance gap versus closed models closed to near-zero on knowledge benchmarks and single-digit points on reasoning by early 2026; self-hosting at scale requires 4–8× H100 GPUs minimum, which changes the cost math entirely. Reasoning models (o3, DeepSeek-R1, Claude Opus with extended thinking, Gemini Deep Think): trained with reinforcement learning to generate long internal chain-of-thought traces before answering; 3–10× slower and costlier per task but dramatically better at math (AIME), science, and multi-step code — use them where the complexity justifies the latency. DeepSeek-R1 in January 2025 proved reasoning capability can emerge from pure RL without supervised fine-tuning, at ~96% lower API cost than o1 at launch — a result that bent the cost curve for the whole category. See module 4 for the decision framework.

A worked cost example

Before moving on, here is what the loop costs in practice.

Suppose you are building a support chatbot. Average conversation: 10 turns, 200 tokens of user input per turn, 300 tokens of model output per turn. System prompt: 500 tokens, re-sent every call.

Per call:
  Input tokens  = system_prompt(500) + history(grows each turn) + user_msg(200)
  Output tokens = model_response(300)

Turn 1 input:  500 + 0 + 200    = 700 tokens
Turn 5 input:  500 + 4×500 + 200 = 2,700 tokens  (history = 4×(200+300))
Turn 10 input: 500 + 9×500 + 200 = 5,200 tokens

Total input across 10 turns: sum of turns 1-10 ≈ 29,500 tokens
Total output across 10 turns: 10 × 300 = 3,000 tokens

At a mid-tier model ($3/$15 per M input/output tokens, illustrative):
  Input cost:  29,500 / 1M × $3  = $0.089
  Output cost: 3,000 / 1M × $15  = $0.045
  Total per conversation: ~$0.13

At 10,000 conversations/day: ~$1,300/day → ~$40K/month

That is not alarming for a supported product, but it scales linearly with usage and quadratically with conversation length. This is why prompt caching (cache the stable system prompt prefix, pay ~10% for cache hits) and token cost optimization are first-week production decisions at any volume.

Where to go next

The prediction loop is the foundation. Everything else in this course is about working with it intelligently:

• Transformers Without the Math — what is actually happening inside that forward pass: how tokens become vectors, why attention is the key operation, and what transformers can and cannot represent
• Tokens, Costs, and the Context Window — the full cost model with numbers; why context accumulates; how to manage the window without degrading quality
• The Model Landscape: Choosing Without Getting Played — a decision framework for picking between closed, open-weight, and reasoning models for a given task
• Tokenization and BPE — the deep dive on how BPE works, why non-English inputs cost more, and where tokenization causes subtle production bugs
• Decoding and Sampling — full treatment of temperature, top-k, top-p, min-p, and speculative decoding with worked examples
• Why Models Hallucinate — the mechanistic explanation for why hallucination is structural, not a gap to be closed with a better system prompt
• Capability Limits and Failure Modes — what the prediction loop cannot do, and which failure modes require system-level solutions rather than model improvements