Embeddings and positional encoding: how tokens become geometry

In 2023, Liu et al. ran a simple experiment: put the answer to a question at different positions inside a long prompt and measure retrieval accuracy. The result, published as "Lost in the Middle", was a U-shaped curve. Facts placed at the beginning or end of the context were retrieved reliably; facts placed in the middle sometimes did worse than giving the model no document at all. Every sequence fit comfortably inside the models' context windows. The problem was not the context limit — it was that information in the middle of a long sequence receives systematically less attention than information at the edges. That "lost in the middle" effect traces directly back to how positional encodings and attention interact. You cannot reason about it without understanding what happens before attention even runs.

From integer to vector: the embedding table

After tokenization, every piece of text is a list of integers. The word "astronaut" might be token 47210. That integer carries no semantic information — 47210 and 47211 are adjacent but could mean completely unrelated things. The job of the embedding table is to map each integer to a vector that does carry meaning.

The table is a matrix of shape vocab_size × d_model. For Llama 3 8B: 128,256 × 4,096. Each row is the embedding for one token. Forward pass through this layer is literally an index operation: look up row 47210, return that 4,096-dimensional float vector. No multiplication happens at this step.

The table is learned during training. The loss signal from next-token prediction flows backward through every layer, including through the embedding lookup, and the row vectors adjust until the geometry that emerges helps prediction. Semantically similar tokens end up with similar vectors not because someone designed it that way, but because similar tokens appear in similar contexts and the model learns to predict those contexts from the same vector.

import torch
import torch.nn as nn

class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size: int, d_model: int):
        super().__init__()
        # This is the entire embedding layer: just a lookup table
        self.table = nn.Embedding(vocab_size, d_model)

    def forward(self, token_ids: torch.Tensor) -> torch.Tensor:
        # token_ids: [batch, seq_len]  → returns [batch, seq_len, d_model]
        return self.table(token_ids)

# Llama 3's 128,256-token vocab at d_model=4096
emb = TokenEmbedding(128_256, 4_096)
print(f"Parameters: {emb.table.weight.numel():,}")   # 525,336,576
print(f"Memory (bf16): {emb.table.weight.numel() * 2 / 1e9:.2f} GB")  # ≈ 1.05 GB

The geometry that emerges has real structure. The classic demo — king - man + woman ≈ queen — is a consequence of this structure. But the space is high-dimensional and the analogies are noisy; do not over-extend the geometric metaphor when thinking about production behavior.

Weight tying shares the embedding table with the output projection head (the matrix that converts the final hidden state back into a probability distribution over the vocabulary), halving the parameter count for vocabulary-related weights. It is standard where that halving actually matters: small models like GPT-2, Gemma, and Llama 3.2 1B/3B, where the vocab matrices are a large fraction of total parameters. Most production models at 7B and above — Llama 3 8B/70B, Mistral, Qwen 7B+ — keep the input and output matrices separate; at that scale the memory saving is marginal and the two matrices benefit from learning different geometry.

The order problem

Attention is a set operation. Given a set of vectors, it computes for each vector a weighted sum of all others, where the weights depend on content similarity. This is exactly what makes it powerful — it can pull together any two tokens regardless of distance. But it also means the same set of tokens produces the same output regardless of their order. Feed the model ["the", "dog", "bit", "the", "man"] in any permutation and the content-only attention weights are the same.

That is catastrophically wrong for language. Positional encoding is the only thing that breaks the symmetry.

flowchart TD
    A["Token vectors (no position)"]
    B["Attention: computes similarity\nbetween all pairs"]
    C["Output: weighted blend\nof all tokens"]
    A --> B --> C
    NOTE["Without positional encoding:\n'dog bit man' = 'man bit dog'\nto the attention mechanism"]
    style NOTE fill:#ff2e88,color:#fff

Absolute positional embeddings: the original approach

The first generation of large language models — including GPT-2 — handled this with learned absolute positional embeddings. Alongside the token embedding table, there is a second table of shape max_seq_len × d_model. Each position index (0, 1, 2, …) maps to a vector, and that vector is added to the corresponding token embedding before the sequence enters the transformer blocks.

The vectors are learned during training. Position 0 develops one characteristic signature, position 1 another, and so on up to the maximum trained length.

This works. It is simple. The problem is max_seq_len. If the model trained on sequences up to length 2,048, it has learned embedding vectors for positions 0–2047. Position 2048 does not exist in the table. You cannot run inference on a longer sequence without retraining or extrapolating in some way.

The original "Attention Is All You Need" paper proposed sinusoidal embeddings as an alternative: instead of learning position vectors, compute them analytically using a family of sine and cosine functions at different frequencies. The formula guarantees that relative distances have consistent representations regardless of absolute position, which allows limited extrapolation beyond the training length. Sinusoidal embeddings fell out of favor as training budgets grew — learned embeddings simply fit the training distribution better when you have the compute to train on long sequences.

RoPE: the current standard

Rotary Positional Embedding (RoPE), introduced in the 2021 RoFormer paper and now the default in every major open-weight model as of 2025, takes a different approach. Instead of adding a position vector to the token embedding, RoPE applies a rotation to the query (Q) and key (K) vectors inside each attention head, with the rotation angle proportional to the position.

flowchart LR
    V["Token vector\n[v₁, v₂, ..., v_d]"] --> SPLIT["Split into\n2D subspaces\n(v₁,v₂), (v₃,v₄), ..."]
    SPLIT --> ROT["Rotate each pair\nby θ · position\n(different θ per subspace)"]
    ROT --> QK["Rotated Q and K\nfor attention score"]
    style ROT fill:#0e7490,color:#fff

The key mathematical property: the dot product between a rotated Q at position m and a rotated K at position n depends only on m - n, the relative distance between them. The absolute positions m and n cancel out. This is not a design choice that happens to work — it is guaranteed by construction.

Why does this matter? Because it means the attention mechanism naturally encodes relative relationships. Token 7 attending to token 3 sees the same relative distance signal (offset 4) regardless of whether those tokens appear in a 512-token or a 128K-token context. This is what allows RoPE-based models to generalize to longer sequences than they trained on, at least within a range.

The rotation uses different angular frequencies for different 2D subspace pairs in the embedding: early pairs rotate slowly (low frequency, capturing long-range position), later pairs rotate quickly (high frequency, capturing short-range position). This multi-resolution structure is the same intuition behind sinusoidal embeddings, now applied inside the attention computation rather than as a pre-embedding step.

import torch
import math

def rotate_half(x: torch.Tensor) -> torch.Tensor:
    """Rotate pairs of dimensions: [x1, x2] → [-x2, x1]"""
    x1, x2 = x[..., ::2], x[..., 1::2]
    return torch.stack([-x2, x1], dim=-1).flatten(-2)

def apply_rope(q: torch.Tensor, k: torch.Tensor, positions: torch.Tensor,
               base: float = 10_000.0) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Apply Rotary Position Embedding to Q and K.
    q, k: [batch, heads, seq_len, head_dim]
    positions: [seq_len]  (usually just torch.arange(seq_len))
    """
    head_dim = q.shape[-1]
    # One frequency per pair of dimensions
    dim_idx = torch.arange(0, head_dim, 2, dtype=torch.float32, device=q.device)
    theta = 1.0 / (base ** (dim_idx / head_dim))        # [head_dim/2]
    angles = torch.outer(positions.float(), theta)       # [seq_len, head_dim/2]
    cos = torch.cos(angles).repeat_interleave(2, dim=-1) # [seq_len, head_dim]
    sin = torch.sin(angles).repeat_interleave(2, dim=-1)

    q_rot = q * cos + rotate_half(q) * sin
    k_rot = k * cos + rotate_half(k) * sin
    return q_rot, k_rot

This is a simplified version for illustration; production implementations like transformers use optimized CUDA kernels and handle grouped-query attention (GQA) where K and V are shared across head groups. The self-attention article covers GQA in detail.

Context extension: YaRN, LongRoPE2, and the honest limits

The original RoPE training uses a fixed base frequency (base=10_000 in the paper; Llama 3 uses base=500_000 for better long-context behavior). At inference time, positions beyond the training length see rotation angles outside the range the model learned. The model does not crash — but recall degrades.

RoPE scaling modifies the frequencies at inference time (or fine-tuning time) to fit a longer range into the same rotation geometry. The simplest method just multiplies all frequencies by a scaling factor s: a 4x scale maps positions 0–32K into the rotation range that 0–8K occupied during training. This works but unevenly — different frequency components generalize differently, so uniform scaling hurts short-range relationships to help long-range ones.

YaRN (Yet another RoPE extensioN) and LongRoPE2 address this by treating different frequency dimensions separately. Low-frequency dimensions (which encode long-range distance) get scaled up; high-frequency dimensions (which encode short-range, positional fine detail) are left closer to their trained range. This preserves local coherence while extending the effective context.

RoPE scaling: order-of-magnitude expectations

Training length   → Eval length   → Expected recall degradation
8K               →  8K            → Baseline
8K               → 16K (2x)       → Minor; most production models handle this well
8K               → 32K (4x)       → Moderate; YaRN recovers most quality with fine-tuning
8K               → 64K (8x)       → Significant; requires targeted long-context fine-tuning
8K               → 128K (16x)     → High risk; verify with needle-in-haystack on your data

The business-relevant takeaway: when a model provider says "we extended the context window from 8K to 128K," look for whether they fine-tuned on long-context data or just applied a frequency rescaling. A pure inference-time RoPE rescaling at 16x is speculative. Fine-tuned extension to 16x on dense long-context data is much more trustworthy — but still needs your own evaluation before production use. The context windows article covers how KV cache memory interacts with these claims.

You can feel the failure mode directly in the visualizer below: drag the relevant fact along the context strip and watch retrieval accuracy trace the same U-shaped curve Liu et al. measured — strong at the edges, sagging in the middle.

{ "type": "lost-middle", "title": "Recall vs position: the lost-in-the-middle effect" }

What the embedding geometry actually looks like

After training, the embedding space has real structure, and that structure affects production behavior in non-obvious ways.

Tokens that appear in similar contexts — synonyms, related verbs, tokens in the same grammatical role — cluster together. Tokens that appear in opposite contexts sit apart. The geometry is not flat: there are local neighborhoods, rough semantic clusters, and directional patterns. But this structure is a byproduct of next-token prediction on a specific training corpus, which means it reflects the distribution of that corpus, including its biases and gaps.

Two practical consequences that come up regularly:

First, rare tokens have poor embeddings. If a token appears very infrequently in training data, its embedding row did not receive many gradient updates and sits in a part of the space that is only weakly constrained by anything. This is one reason models behave strangely on rare proper nouns or domain-specific jargon — the embedding is barely trained, not just unknown.

Second, cross-lingual quality gaps. The embedding table learns good geometry for tokens that appear frequently. English tokens appear far more often in most training datasets than tokens from lower-resource languages. The geometric relationships in English embedding space are denser and more structured than in, say, Yoruba or Amharic. Combined with the fertility problem covered in tokenization — non-Latin scripts producing 2–3x more tokens per word — this is a double cost for non-English workloads.

Embedding dimensions across production models

The embedding dimension d_model grows with model size. Roughly:

Model class	d_model	Embedding table size (128K vocab, bf16)
Small (~1B)	2,048	~525 MB
Medium (~7–8B)	4,096	~1.05 GB
Large (~70B)	8,192	~2.1 GB
Very large (~405B)	16,384	~4.2 GB

These are not the bottleneck — the attention and MLP weights dwarf them. But they matter when planning vocabulary size. Modern tokenizers run 128K–256K tokens; a 256K vocabulary at d_model=8192 puts the table at ~4.2 GB, doubled again if the output head is untied. Vocabulary size has become a genuine design decision, not a footnote.

The interaction with context windows and KV cache is worth flagging: d_model and the number of attention heads determine the KV cache size per token. Larger d_model means larger KV entries, which means more memory per token of context.

{ "type": "kv-cache", "model": "8b", "context": 32000, "title": "Memory split: model weights vs KV cache at 32K context" }

What breaks

The marketed window vs effective recall

This is the most practically important failure mode in this section. A model with a 128K context window has been trained or fine-tuned to accept 128K tokens. That is not the same as reliably retrieving information placed anywhere in that window. Needle-in-haystack evaluations — where a specific fact is placed at a random position in a long filler document and the model must retrieve it — consistently show degraded recall for content placed in the middle of a long context.

The mechanism is related to positional encoding: the high-frequency RoPE dimensions that encode short-range relationships are well-trained; the low-frequency dimensions that encode very long-range relationships may have seen few training examples. The result is that distant positions are harder to distinguish and attend to. Attention also concentrates on attention sinks — certain tokens (beginning-of-sequence tokens, punctuation) that absorb disproportionate attention mass regardless of their content relevance, a mechanism documented in 2023 in the StreamingLLM work on attention sinks. Both effects push information in the middle of a long context into lower-attention territory.

For production systems that depend on retrieving specific content from long contexts, benchmark your actual model on your actual content distribution before committing to full-context approaches. Long context vs RAG works through when retrieval-augmented approaches outperform context stuffing — the short version is that for locatable information, RAG usually wins on both recall and cost.

Fine-tuning does not fix sparse embeddings

If you fine-tune a model on domain-specific data, the embedding table updates only for tokens that appear in the fine-tuning data. Rare tokens in your domain may still have poor embeddings if they did not appear in pre-training and also do not appear in your fine-tuning set. The usual mitigation is vocabulary extension — adding new tokens to the table with initialized embeddings — but this adds complexity and risks destabilizing adjacent rows.

RoPE base matters for long context

The choice of base frequency in RoPE (10_000 in the original paper, 500_000 in Llama 3, higher in some recent models) affects how the rotation frequencies are distributed across dimensions. A higher base spreads the frequencies out more, giving more headroom for long-range position encoding. When you see models with different context windows and similar architectures, the RoPE base is often the key variable. This is not exposed in most inference APIs, but it matters when fine-tuning or choosing a base model for context extension.

The embedding layer in the forward pass

To make the above concrete, here is how embedding and positional encoding fit into the full transformer forward pass:

sequenceDiagram
    participant IN as Input text
    participant TOK as Tokenizer
    participant EMB as Embedding table
    participant BLOCK as Transformer blocks (×N)
    participant OUT as Output head

    IN->>TOK: "the dog bit the man"
    TOK->>EMB: [464, 3290, 3253, 262, 582]
    EMB->>BLOCK: 5 × d_model vectors (no position yet)
    Note over BLOCK: RoPE applied inside each\nattention head during Q,K rotation.\nAll N blocks share the same positional\nlogic — no separate PE layer.
    BLOCK->>OUT: contextualised representations
    OUT-->>IN: next-token distribution

In RoPE-based models, there is no separate positional encoding layer visible in the architecture diagram. The embedding table outputs plain token vectors. Position is injected inside each attention block when Q and K are rotated before the dot product. The transformer blocks (covered in detail in the transformer block article) each apply their own RoPE rotation with the same base frequencies — position is re-applied at every layer.

The design decision in practice

When you are choosing a base model for a production system, the positional encoding design affects three concrete decisions:

Context length needed vs context length reliable. If your task genuinely requires retrieving facts spread across 64K tokens of text, benchmark the model on that task. Do not assume the marketed window length represents reliable recall. Most tasks that seem to need very long context can be reformulated as retrieval over a structured index — which is faster, cheaper, and more reliable than full-context attention.

Context extension via fine-tuning. If you need to extend a model beyond its training context — common when adapting a 4K-context base model to a 32K production use case — RoPE scaling with YaRN-style frequency adjustment is the standard technique, but it requires fine-tuning on actual long-context examples to recover quality. Do not trust inference-time-only scaling at more than 2x the training length without your own benchmarks.

Vocabulary and embedding table size. For specialized domains with unusual terminology (medical, legal, code in non-mainstream languages), consider whether the tokenizer's vocabulary covers your domain well. Frequent out-of-vocabulary decompositions (your key term tokenizes to 4–6 subword tokens) increase costs, decrease quality, and indicate that the embedding table has sparse coverage of your domain. This is upstream of retrieval quality and model performance — something to investigate in the tokenization article and the embedding model choice guide.

The geometry of embedding space and the structure of positional encoding set the floor for everything the model does afterward. Every attention pattern, every decoding decision, every context-window tradeoff that comes up in self-attention and decoding strategies runs on top of the representations built here. Getting a mental model of this layer right is the straightest path to not being surprised by your model's behavior in production.