Transformers Without the Math

You don't need to derive backpropagation to drive this thing. But you do need to know what the engine does when you floor it — because the failure modes aren't random. They follow directly from the architecture. When your model ignores a fact you put in the prompt, when a 128K-context call costs ~30× more than a 4K one, when fine-tuning changes tone but not knowledge — that's the transformer doing exactly what it's designed to do. This module builds the mental model that makes those behaviors predictable.

No matrix multiplications. No gradient flows. Just the three concepts that actually govern what these models can and can't do: how tokens become vectors in a geometric space, how attention routes information, and how depth stacks understanding.

Step 1: Text becomes geometry

A language model doesn't read text. It reads integers. The first job of the pipeline is to turn your string into a sequence of token IDs, and those IDs into vectors.

{ "type": "tokenizer", "title": "Your text as the model sees it" }

The tokenizer — covered in detail at Tokenization and BPE — turns words into subword chunks. "transformers" might become ["transform", "ers"], two tokens. Each token ID then gets looked up in an embedding table: a matrix with one row per vocabulary entry, each row a vector of 768 to 8192 floating-point numbers depending on the model.

Here's the key claim: these vectors aren't random assignments. They're learned end-to-end so that tokens that appear in similar contexts land near each other in this high-dimensional space. "king" and "queen" cluster together. "Paris" and "France" sit near each other in a way that reflects their relationship. You can literally subtract the "male-ness" direction from "king" and add "female-ness" and land close to "queen." That's not magic — it's what happens when you train a neural network to predict what word comes next across a trillion tokens.

But vectors don't carry position. The embedding table tells the model what each token means, not where it appears in the sequence. "Dog bit man" and "Man bit dog" would produce the same set of vectors in an arbitrary order without positional encoding — some mechanism that tells the model where each token sits in the sequence.

Older models used fixed sinusoidal patterns (the original 2017 Transformer) or a learned absolute-position table added to the embeddings (GPT-2 style). Every production model since 2023 uses Rotary Positional Embedding (RoPE), which encodes position as a rotation applied to the Query and Key vectors inside every attention layer — nothing gets added to the input embeddings at all. The elegant property: RoPE naturally captures relative distance between tokens, not just absolute position. That structure is what makes the context-extension tricks below possible — but out of the box it does not generalize past the training length; push a vanilla RoPE model beyond the sequence lengths it trained on and recall falls apart. Llama 3, Gemma 2, Qwen 2.5, Mistral — RoPE across the board.

One pitfall worth knowing: when vendors extend a model's context window past its training length (via YaRN or LongRoPE2 scaling), they're essentially extrapolating the positional encoding. A 2× extension is generally reliable; 8× requires careful fine-tuning or recall degrades badly. A model claiming a 128K context window may have been trained primarily on shorter sequences with RoPE scaling applied late — effective recall can collapse past 32K even if the window accepts more.

flowchart LR
    T["Raw text"] --> TOK["Tokenizer\n(BPE)"]
    TOK --> IDS["Token IDs\n[45, 1203, 7, ...]"]
    IDS --> EMB["Embedding table lookup"]
    EMB --> VEC["Vectors\n(d_model dims)"]
    VEC --> TF["Transformer layers"]
    POS["RoPE: rotates Q/K inside\neach attention layer"] --> TF
    style T fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f
    style TF fill:#a855f7,stroke:#a855f7,color:#fff

Step 2: Attention — every token votes on what matters

This is the mechanism that made transformers work where everything before had struggled. The idea is almost embarrassingly simple stated plainly: let every token look at every other token and decide how much to pull from each.

Each token produces three vectors from its embedding:

• Query (Q): what am I looking for?
• Key (K): what do I offer?
• Value (V): what information do I carry?

For every pair of tokens, the model computes a similarity score between the first token's Query and the second token's Key. That score — after scaling and a softmax to make it a probability — determines how much of the second token's Value gets blended into the first token's output. Run this across all positions simultaneously, and you've computed one attention head.

One constraint matters more than any other detail here: the decoder-only models you actually use apply a causal mask — each token attends only to itself and earlier positions, never ahead. "Bank" can't peek at "deposit" three words later; instead, "deposit" does the backward-looking blending when its turn comes. This isn't a limitation bolted on for training convenience — it's what makes the KV cache (below) possible at all, because a past token's K and V never change once computed.

{ "type": "attention", "sentence": "bank-river", "title": "Watch tokens read each other" }

That widget makes concrete what a single head computes. The word "bank" in "I went to the bank by the river" attends heavily to "river" — which is how the model resolves that "bank" means riverbank, not financial institution. It's doing this with the attention weights, not by looking up a dictionary.

A single head is limiting. Real models use multi-head attention: 32, 40, 64 independent Q/K/V projections running in parallel, each attending to different aspects of the input. Some heads specialize in tracking syntax (linking subjects to verbs). Others track coreference (tying "it" back to "the animal" five sentences earlier). Others attend mostly to nearby context. This specialization isn't explicitly programmed — it emerges from training.

The quadratic cost problem

Here's the expensive part. If you have n tokens, you're computing a score for every pair — that's n² scores. Double the context, quadruple the attention work. At 4K tokens you compute 16 million scores per head per layer. At 128K tokens that's 17 billion — roughly 1,000× the attention work for 32× the tokens. Total prefill compute for a 70B model lands closer to 100×, because the MLP layers (most of the FLOPs at short context) scale linearly. And your API bill grows only ~30×, because providers price per token. Three different curves — 30× dollars, 100× total compute, 1,000× attention scores — all from the same n² root cause.

n = 4,096 tokens   → n² = 16.7M  scores per head
n = 32,768 tokens  → n² = 1.07B  scores per head
n = 131,072 tokens → n² = 17.2B  scores per head

Flash Attention (v2/v3) rewrites the kernel to avoid materializing the full n×n matrix in GPU memory, computing it in tiles instead. It doesn't change the theoretical complexity but cuts the memory cliff dramatically — every serious inference deployment uses it now.

GQA: the memory optimization behind modern models

There's a second cost: the KV cache. During autoregressive generation (one token at a time), the model needs the K and V tensors for all previous tokens to compute attention. Recomputing them every step would be prohibitive, so they're cached.

At long context the KV cache dominates memory. A 70B model using standard multi-head attention at 128K context would need over 300 GB just for K and V — more than most GPU clusters have available for a single request. Grouped-Query Attention (GQA) solves this by sharing K and V projections across groups of Q heads. If 8 query heads share one K and one V, the cache shrinks 8×. Quality loss is negligible in practice — which is why GQA is the universal default in every major 2025-2026 open-weight model.

{ "type": "kv-cache", "model": "70b", "context": 32768, "title": "Where your GPU memory goes" }

Move the sliders. Notice what dominates at long context. That's the practical reason GQA exists and why context length has a real cost ceiling even on large GPU clusters.

Step 3: The transformer block — stacking understanding

One attention operation is interesting. Forty-eight stacked layers is what makes GPT-4-class models work. To understand why depth helps, you need to understand how the layers connect.

flowchart TD
    RS0["Residual stream\n(token's running vector)"]
    RS0 --> LN1["LayerNorm"]
    LN1 --> ATT["Multi-head attention"]
    ATT --> ADD1["+ (add back)"]
    RS0 --> ADD1
    ADD1 --> RS1["Updated residual stream"]
    RS1 --> LN2["LayerNorm"]
    LN2 --> MLP["Feed-forward MLP\n(expand 4×, contract)"]
    MLP --> ADD2["+ (add back)"]
    RS1 --> ADD2
    ADD2 --> RS2["Output residual stream"]
    style RS0 fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f
    style RS2 fill:#a855f7,stroke:#a855f7,color:#fff

The key architectural insight is the residual stream. Each token has one vector that persists across all layers. The attention sublayer reads from this vector and produces an update — a delta — that gets added to the stream. The MLP sublayer does the same. Neither replaces the vector; both nudge it.

Think of it as a shared whiteboard. The original token embedding is the first thing written. Each layer adds notes, context, and refinements. By layer 48, the vector for the word "bank" has been updated by every attention and MLP layer — it now contains a rich blend of its own meaning, what it read from neighboring tokens, and what those tokens read from their neighbors. The original signal is still there, contributing; it's just been enriched by 48 rounds of context.

This additive structure is why very deep models work at all. Each layer can make small, targeted improvements without destroying what earlier layers computed. It's also why fine-tuning with LoRA (Low-Rank Adaptation) works: LoRA injects small update matrices at the Q, K, V, and MLP projections — the exact "write" operations in the residual stream — without touching the base weights.

What attention and MLP each contribute

Research in mechanistic interpretability has found that these two sublayers do qualitatively different things:

Attention moves information. It pulls relevant context from across the sequence into each position's vector. The "it" in "the animal didn't cross the street because it was too tired" needs to pull the meaning of "animal" to resolve the pronoun. That's attention.

The MLP stores knowledge. The feed-forward sublayer expands the vector dimension by ~4× (via a SwiGLU gate in Llama 3, Gemma), applies a nonlinearity, then contracts back. Research has found that ablating specific MLP neurons removes specific facts from the model's outputs — particular neurons "know" that Paris is in France, or that the boiling point of water is 100°C. The MLP is the model's fact store.

The two are complementary. Attention without depth gives you good context routing but shallow pattern matching. Depth without attention gives you a model that can't relate distant tokens. Every production model stacks 32 to 96 blocks of the combined pair.

Step 4: From vector to word

The stack of layers has to end somewhere, and it ends undramatically. After the final block, the last position's residual vector — 8,192 numbers for a 70B-class model — gets multiplied by an unembedding matrix (often the embedding table reused in reverse), producing one score per vocabulary entry: roughly 128K logits for a Llama 3 tokenizer. A softmax turns those scores into a probability distribution. If the context so far is "The capital of France is", the token " Paris" might hold 97% of the mass, with " a", " located", and " the" splitting most of the rest. The model then samples one token from that distribution — greedily taking the maximum at temperature 0, with controlled randomness above it.

That's the entire output of a forward pass: one token. To get the next one, the sampled token is appended to the sequence and the whole stack runs again — and this loop is where the KV cache from Step 2 earns its keep, because the new pass only computes Q, K, and V for the single new token and attends against everything already cached. A 500-token answer is 500 forward passes. Predict, sample, append, repeat: that's autoregressive generation, and it's why output tokens dominate your latency while input tokens dominate the prefill compute. How the sampling knobs — temperature, top-p — shape which token gets picked is covered in Tokens, Costs, and the Context Window.

What breaks in practice

Understanding the architecture explains the failure modes — none of them are mysterious once you have the mental model.

The lost-in-the-middle problem. Attention is uniform in principle — every token can attend to any earlier token, no matter how far back. But in practice, models pay disproportionate attention to the beginning and end of long contexts. Information buried in the middle of a 100K-context window is frequently ignored. Needle-in-haystack benchmarks confirm: many 128K models lose reliable recall past 32-64K, depending on training data distribution. The context window size is a capacity claim, not a recall guarantee.

flowchart LR
    A["128K context window"] --> B["Claimed capacity"]
    A --> C["Effective recall\n~32-64K for many models"]
    B -.vs.- C
    C --> D["Design for effective recall\nnot marketed limit"]
    style A fill:#00e5ff,color:#0a0a0f
    style D fill:#a855f7,color:#fff

Attention sinks. Research from 2023 (Xiao et al., "StreamingLLM") documented that specific tokens — the first token (BOS), certain punctuation — absorb disproportionate attention mass regardless of content relevance. Deep layers "park" attention on these sink tokens, effectively wasting capacity. This is a mechanistic contributor to context-retrieval failures at long context.

Temperature 0 is not deterministic. At greedy decoding (temperature = 0), the model always picks the highest-probability next token. But GPU floating-point operations aren't fully deterministic across different hardware configurations, batch sizes, or kernel versions. If you need reproducibility, you need it at the infrastructure level — not just a temperature setting.

Context length is not knowledge injection. A common production mistake: stuffing long documents into context hoping the model will "learn" the information. It doesn't learn — it reads. And at 500K tokens, the quadratic attention cost is enormous, recall degrades, and a well-tuned RAG pipeline will beat full-context stuffing on every metric that matters: latency, cost, and answer quality. The context window module works through the cost math explicitly.

Fine-tuning changes behavior, not facts. Because knowledge lives in the MLP layers and fine-tuning typically touches Q/K/V and MLP projections with a small adapter, injecting new facts via fine-tuning is unreliable. The model's MLP weights are large, specific, and hard to update cleanly — fine-tuning on new factual data can introduce inconsistencies. Fine-tuning is best used for style, format, instruction-following, and task specialization. For factual grounding, use RAG in One Pass.

The cost of a transformer call, in one estimation

Back-of-the-envelope that every AI engineer should be able to do in their head:

Model: 70B params, GQA (8 groups), d_model = 8192, 80 layers
Context: 32K input tokens → generating 512 output tokens

KV cache memory per token per layer:
  2 × (n_kv_heads × d_head) × sizeof(dtype)
  GQA with 8 KV heads, d_head = 128, fp16:
  = 2 × (8 × 128) × 2 bytes = 4096 bytes per token per layer

Total KV cache: 32,768 tokens × 80 layers × 4096 bytes
             ≈ 10.7 GB for input prefix alone

Model weights: 70B × 2 bytes (fp16) = ~140 GB

Total VRAM at peak: ~151 GB → needs 2× H100 80GB at minimum
(before batching multiple requests together)

That's why a 70B model at 32K context can't fit on a single GPU, and why the inference serving question at long context is almost entirely a memory placement problem. The KV Cache deep dive goes through the full math with GQA and MHA variants.

Where to go next

This module gave you the mental model. Each of these goes substantially deeper:

• Tokens, Costs, and the Context Window — the practical consequences of tokenization and context length on cost and behavior, with worked pricing math.
• Self-Attention: The Mechanism That Reads Every Token Against Every Other — the full Q/K/V mechanics, Flash Attention, and how multi-head specialization emerges.
• The Transformer Block: Residual Stream, MLP, and What Each Part Actually Learns — mechanistic interpretability findings on circuits, induction heads, and what each sublayer actually computes.
• Context Windows, KV Cache, and the Reality Behind 1M Token Claims — why marketed context length and effective recall diverge, and how to size your system against the real number.
• Tokenization and BPE: Why Your Model Cannot Spell Strawberry — why non-English text costs 2-3× more tokens and why arithmetic is hard for a model that reasons at the subword level.
• Embeddings and Positional Encoding: How Tokens Become Geometry — the geometry behind embedding spaces and how RoPE determines how far a model can really see.
• Context Engineering: Filling the Window Right — once you know what the context window is, this covers how to fill it without wasting it.