Tokenization and BPE: Why your model cannot spell strawberry

A demo went wrong at a company that was building a Thai-language customer service bot. They'd designed context management assuming their system prompt plus RAG chunks plus history would fit comfortably in 32K tokens. In production, the Thai text was consuming 2.3x what the English equivalent had during prototyping. The context window was overflowing, the history was being truncated mid-conversation, and the bot was losing track of customer details mid-ticket. The fix was a five-line tiktoken call on actual production data that immediately showed fertility of ~2.7 tokens/word instead of the ~1.2 the English prototype had shown. The real lesson was not "measure things" but "the tokenizer is not neutral infrastructure — it bakes in assumptions about what language looks like."

What the model actually receives

When you send a prompt to any modern LLM, the first thing that happens is tokenization. The model never sees the string "How are you?". It sees something like [4516, 527, 499, 30] — a sequence of integer token IDs drawn from a fixed vocabulary. The transformer then looks up each ID in an embedding table to get a dense vector, and that vector is what the actual computation operates on. Characters, spaces, punctuation — none of that exists inside the model. Only IDs.

This matters because the mapping from text to IDs is lossy in specific ways. Two strings that look different to humans can map to the same tokens. Two strings that look nearly identical can map to very different token sequences. The transformer has no way to inspect the characters inside a token; it can only work with the relationships between tokens that it learned during training.

How BPE builds a vocabulary

Byte Pair Encoding was originally a data compression algorithm from 1994. Its application to NLP (by Sennrich et al., 2016) turned it into the dominant tokenization approach for modern LLMs, with variations used by GPT-4 (cl100k_base), Llama 3, Qwen, Mistral, and most other current models.

The algorithm is straightforward:

1. Start with a base vocabulary of bytes (0–255) or characters.
2. Count every adjacent pair of vocabulary symbols in the training corpus.
3. Merge the most frequent pair into a new vocabulary entry.
4. Repeat until reaching the target vocabulary size (e.g., 100,277 for cl100k).

In practice this means common English words become single tokens early in the merging process. " the" (with a leading space) becomes a single token because it appears billions of times. " strawberry" might stay as [" straw", "berry"] because the full word is less frequent and the merge budget runs out on higher-priority pairs first.

Here's the key insight: the vocabulary is a learned artifact of training data distribution, not a linguistic decision. There is no reason based on English grammar that " the" is one token and " strawberry" is two. It happened because " the" appeared more often in the training corpus.

flowchart TD
    A["Corpus: 'the cat the cat the dog'"] --> B["Char pairs: t+h=3, h+e=3, etc."]
    B --> C["Merge most frequent: 'th'"]
    C --> D["New corpus: 'th e c a t th e c a t th e d o g'"]
    D --> E["Next most frequent: 'the'"]
    E --> F["...repeats until vocab size reached"]
    style A fill:#1e1e2e,color:#cdd6f4
    style C fill:#00e5ff,color:#0a0a0f
    style E fill:#00e5ff,color:#0a0a0f
    style F fill:#0e7490,color:#fff

The result is a vocabulary where:

• Extremely common words are single tokens: " is", " the", " and", " for"
• Common but not ubiquitous words split into recognizable pieces: "token" + "ization", "straw" + "berry"
• Rare words, proper nouns, non-English text, and code symbols fragment further: "ม", "อ", "อ" for Thai characters that each get their own token or share only short merges

{ "type": "tokenizer", "title": "Live BPE tokenizer: type any text to see it split" }

The fertility problem

Fertility is the ratio of tokens to words for a given language or domain. It tells you how efficiently a tokenizer represents your text.

OpenAI's tokenizers — cl100k for GPT-4, o200k for GPT-4o — were trained on corpora that are heavily English. The fertility numbers reflect that:

Language / Domain	Approximate tokens per word	Notes
English	~1.1–1.3	Common words are single tokens; suffixes and rare words split
French / German / Spanish	~1.4–1.7	Longer inflections, some compound words
Russian	~1.5–2.0	Cyrillic script, rich morphology
Arabic	~2.0–3.0	Right-to-left, root-and-pattern morphology, few high-freq merges
Thai	~2.0–3.0	No word boundaries in script, limited corpus representation
Korean	~1.5–2.5	Syllable blocks fragment into multiple tokens
JSON (dense keys)	~1.3–1.6	Repeated structural characters cost extra
Python code	~0.9–1.3	Depends heavily on whether tokenizer was trained on code

These numbers are illustrative but grounded in published analyses; your specific numbers depend on the tokenizer version and your text distribution. The practical exercise: call tiktoken.encoding_for_model("gpt-4o").encode(sample_text) on a thousand examples of your real data and plot the distribution.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

samples = {
    "English": "The customer submitted a support ticket about billing.",
    "Thai": "ลูกค้าส่งตั๋วสนับสนุนเกี่ยวกับการเรียกเก็บเงิน",   # same sentence
    "JSON": '{"customer_id": "c_123", "ticket_type": "billing", "status": "open"}',
}

for lang, text in samples.items():
    tokens = enc.encode(text)
    word_count = len(text.split())
    fertility = len(tokens) / max(word_count, 1)
    print(f"{lang}: {len(tokens)} tokens, fertility={fertility:.2f}")

# English: 9 tokens, fertility=1.12
# Thai:    20 tokens, fertility=20.00  (Thai has no spaces → word_count=1, misleading)
# JSON:    22 tokens, fertility=3.67

That Thai line is particularly instructive. Thai has no spaces between words, so word count is a bad denominator. But the raw token count is what matters for cost and context: 20 tokens to express the same information as 9 English tokens. A 128K context window full of Thai content holds roughly 58,000 semantic words instead of ~107,000 — roughly a 2x capacity hit. (On the older cl100k tokenizer the same Thai sentence costs 39 tokens — o200k roughly halved Thai fertility, which is exactly the kind of version drift you should re-measure after a model upgrade.)

Why numbers shatter

Type 42 into a tokenizer: single token. Type 4,200: probably two tokens — "4" and ",200" or some similar split. Type 1,234,567: four or five tokens, fragmented wherever the training data had those digit sequences appearing as distinct sub-words.

The model is not seeing the number 1,234,567. It is seeing a sequence of tokens, each of which maps to a learned vector in embedding space. Those vectors encode statistical co-occurrence patterns from training data, not the arithmetic properties of the digits. When you ask the model to add two numbers, it is doing something more like pattern-completion from training examples than executing an algorithm over digits.

Token splits for numbers (cl100k_base, actual encoder output):
"42"        → [2983]                  (1 token)
"420"       → [12819]                 (1 token)
"4200"      → [12819, 15]             (2 tokens: "420" + "0")
"1234"      → [4513, 19]              (2 tokens: "123" + "4")
"1,234,567" → [16, 11, 11727, 11, 19282] (5 tokens: "1", ",", "234", ",", "567")

This is why "how many Rs are in strawberry" fails. The model sees tokens, and within those tokens, the letter r is not a separately inspectable unit. It would have to reconstruct the character sequence from statistical knowledge of what characters typically compose each token — an unreliable indirect inference.

For tasks that require character-level or digit-level precision, the structural remedy is to break the word or number into characters explicitly at the prompt level:

word = "strawberry"
spelled_out = " ".join(list(word))  # "s t r a w b e r r y"
prompt = f"How many times does the letter 'r' appear in: {spelled_out}"

This works because each character now becomes its own token (or nearly so), making the counting tractable. It is also three times longer and eats context. That is the trade-off.

The tokenization multiplicity bug

Here is a production bug that trips up engineers who build pipelines that construct prompts by concatenating strings:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

# Tokenize pieces separately
piece1 = enc.encode("Hello")
piece2 = enc.encode(" world")

# Tokenize together
combined = enc.encode("Hello world")

print(piece1 + piece2)  # [13225, 2375]   — "Hello" + " world"
print(combined)         # [13225, 2375]   — same here, but not always

# The dangerous case: leading space sensitivity
enc.encode("token")        # [10346]
enc.encode(" token")       # [6602]   — different ID for leading space!
enc.encode("first token")  # [7743, 6602]  — " token" is merged with space

This is not a legacy GPT-2 quirk. Every current GPT tokenizer's pre-tokenizer attaches the leading space to the word, so "token" and " token" are permanently different tokens — the IDs above are live o200k output. The rule that follows: always tokenize your full prompt as a unit. If you tokenize the system prompt, user message, and RAG chunks separately and count tokens independently, then add them up, you may get a different count than tokenizing the assembled prompt as a whole. The differences are usually small but they exist, and when you are managing tight context budgets they can cause off-by-one truncation bugs.

sequenceDiagram
    participant P as Prompt builder
    participant T as Tokenizer
    participant M as Model

    Note over P: BAD: count pieces separately
    P->>T: encode(system_prompt)  → 450 tokens
    P->>T: encode(rag_chunk)      → 200 tokens
    P->>T: encode(user_msg)       → 80 tokens
    Note over P: Assumes total = 730 tokens
    P->>M: send concatenated prompt
    Note over M: Actual token count: 727 (merges across piece boundaries)

    Note over P: GOOD: count the assembled prompt
    P->>P: assemble full prompt string
    P->>T: encode(full_prompt)    → 727 tokens (correct)
    P->>M: send if within budget

Two more caveats before you trust any local count. Chat APIs wrap each message in template tokens — role markers and message separators — so the billed prompt runs a few tokens per message higher than the encoded string; a 40-message history quietly picks up 150+ tokens of overhead that no client-side encode() call will show you. Calibrate against the usage field the API returns (or a count-tokens endpoint where the provider offers one) rather than trusting client-side arithmetic alone. And tiktoken only speaks OpenAI: Claude and Gemini use different tokenizers, so tiktoken counts for those models are approximations — fine for rough budgets, not for exact truncation logic.

Code tokenization: why it matters for model quality

Code tokenizers trained on code-heavy corpora (like the models behind GitHub Copilot, DeepSeek-Coder, or Qwen2.5-Coder) produce more efficient token sequences for common programming patterns. A generic English tokenizer seeing => (arrow function in JavaScript) might produce two tokens: "=" and ">". A code-trained tokenizer merges them to one because => appears millions of times in its training corpus.

This is not just a cost issue. More efficient tokenization means:

1. The same code fits in fewer tokens → more code fits in context.
2. Common operator patterns are single semantic units → the model learns their meaning more reliably.
3. Indentation, brackets, and keywords that appear as single tokens carry richer positional information.

If you are building a code-heavy application and routing through a generic LLM API, check whether the model's tokenizer was trained with code representation. For the major frontier models (GPT-4o, Claude Sonnet, Gemini) this is largely handled for you — their tokenizers include substantial code coverage. For smaller or specialized open-weight models, it matters more.

What breaks

Arithmetic and counting

As described above, numbers shatter across tokens. The model cannot inspect digit-level content of a token. Multi-step arithmetic on large numbers fails not because the model lacks math knowledge but because it never sees the digits as addressable units. Mitigation: use tool calling to route arithmetic to a code interpreter, not the model's generation path. See tool and function calling for the pattern.

Non-English context budget overruns

You design a RAG system with 20 chunks × 200 tokens each = 4,000 token budget for context. The estimate was done in English. The actual data is Arabic, at 2.5 tokens/word. Your "200 token" chunks are actually 500 tokens each. The 4,000-token RAG budget is gone after 8 chunks. Production context overflows.

Fix: measure fertility on your actual corpus. Budget tokens, not words or characters. Store token counts at chunk ingestion time so you can fill context windows correctly without recomputing at query time.

Prompt injection via tokenization edge cases

A less obvious failure: an adversarial user submits text that includes special tokens or token boundary confusion to try to break the prompt template. The classic example is injecting characters that the tokenizer treats as separator tokens (like <|endoftext|> in older GPT tokenizers) to confuse prompt structure. Modern APIs sanitize special tokens, but the attack surface is worth understanding when you are doing custom template assembly. See prompt injection and jailbreaks for the broader attack taxonomy.

Fine-tuning data in a different tokenizer

If you train a model with one tokenizer and deploy with another (or fine-tune a model whose tokenizer you replace), the embedding table is misaligned with the learned token statistics. This is not a common mistake with off-the-shelf fine-tuning APIs, but it shows up when engineers try to extend vocabularies or graft a new tokenizer onto an existing model. The result is garbled output without an obvious error signal.

Estimating costs from character count

Character count is the wrong unit. You might see "approximately 4 characters per token" as a rule of thumb for English. That rule fails for every other language and for dense structured data. The only reliable cost estimation is to call tiktoken.encode() (or the equivalent for your model's tokenizer) on actual samples and compute the 90th-percentile token count. Round up.

Worked cost estimate for a multilingual support chatbot (illustrative):

Inputs:
  - 50,000 requests/day
  - Average input: 200 tokens (system + history + user message, English-based estimate)
  - Average output: 150 tokens
  - Model: GPT-4o at $2.50/1M input + $10.00/1M output (mid-2026 illustrative)

English estimate:
  Input cost:  50K × 200 × $2.50/1M = $25/day
  Output cost: 50K × 150 × $10.00/1M = $75/day
  Total: ~$100/day → ~$3,000/month

Reality (40% of traffic is Arabic, 2.5x fertility on inputs):
  English input tokens: 30K req × 200 = 6M tokens
  Arabic input tokens:  20K req × 200 × 2.5 = 10M tokens (not 4M)
  Total input tokens: 16M (not 10M) → input cost = $40/day
  Total: ~$115/day → ~$3,450/month

The gap is real, the bill is higher, and this is a modest example.
At 40% Arabic and 2M requests/day, the same $15/day input delta
scales 40x to ~$600/day — about $220K/year.

Byte-level BPE and the move toward language parity

The original BPE implementations worked at the character level, which meant any character outside the training vocabulary became an unknown token (<unk>). GPT-2 introduced byte-level BPE: the base vocabulary is the 256 possible bytes of UTF-8, not a set of characters. Every possible Unicode string is representable — nothing is unknown — because any string can be decomposed into bytes.

This solved the unknown-token problem but did not solve the fertility problem. Non-English scripts still produce longer token sequences because the high-frequency merges in the vocabulary reflect the training corpus distribution. Byte-level BPE is necessary but not sufficient for language parity.

The current research direction (as of 2025–2026) includes:

• Parity-aware BPE: down-sampling English merges during vocabulary construction to leave more budget for non-Latin script merges.
• Byte-level models (successors to MegaByte): operate directly on raw bytes rather than learned sub-word tokens, treating tokenization as an architectural choice rather than a preprocessing step. These reduce the language-imbalance problem but are slower because sequences are longer.
• Unigram language model tokenization: an alternative to BPE that selects a vocabulary by maximizing the likelihood of the training corpus under a unigram language model; tends to produce more balanced multilingual coverage than greedy BPE merging.

None of these has displaced BPE at production scale for major frontier models as of mid-2026, but they inform the next generation of tokenizer design.

The engineer's decision checklist

When you are designing a system that involves tokenization, these are the questions that determine whether you get it right:

flowchart TD
    A["Building an LLM application?"] --> B{"Non-English or\ncode-heavy workload?"}
    B -->|Yes| C["Measure fertility on\nactual corpus samples"]
    B -->|No| D["English estimate ok\nas starting point"]
    C --> E{"Fertility > 1.5?"}
    E -->|Yes| F["Adjust context budgets\nand cost estimates\nfor real fertility"]
    E -->|No| D
    D --> G{"Arithmetic or\ncharacter-counting tasks?"}
    G -->|Yes| H["Route to code interpreter\nvia tool call — not model generation"]
    G -->|No| I{"Assembling prompts\nfrom multiple pieces?"}
    I -->|Yes| J["Tokenize the assembled prompt,\nnot individual pieces"]
    I -->|No| K["Proceed with standard\ncontext budget planning"]
    style C fill:#00e5ff,color:#0a0a0f
    style F fill:#00e5ff,color:#0a0a0f
    style H fill:#a855f7,color:#fff
    style J fill:#a855f7,color:#fff

The judgment call: when does tokenization matter enough to act on?

Most of the time, for English text with a frontier model, tokenization is background infrastructure. You do not need to think about it explicitly. But it becomes a first-class engineering concern in three scenarios:

1. Multilingual workloads. If more than 20% of your traffic is in non-Latin-script languages, your English-based cost and context estimates are materially wrong. Measure fertility on your real distribution. Store token counts at chunk creation time. Build your context-filling logic around tokens, not characters or words.

2. Precision tasks on structured or numeric data. If your application involves arithmetic, counting, or exact string matching inside the model's generation path, tokenization fragmentation is a structural enemy. The engineering fix is to push precision tasks to code interpreter tool calls (see tool and function calling), leaving the model to handle language tasks where pattern completion is appropriate.

3. Tight context budgets. At 128K+ context windows with dense inputs — long conversation history, large RAG chunks, complex tool schemas — tokenization inefficiency compounds fast. Profile your actual prompt distribution to find where tokens are going. Common surprises: repeated JSON field names in tool schemas, verbose system prompts, and RAG chunks from non-English documents that consume 2–3x the expected context.

For everything else: measure once, set a realistic per-request token budget, track token counts in your observability layer (see LLM observability), and move on. The internals of BPE are not something you need to tweak — they are something you need to account for when your data distribution makes the assumptions wrong.

The strawberry problem is not going away. Sub-word tokenization is the right engineering trade-off for current transformer architectures, and the alternative (character-level or byte-level models) pays a significant throughput penalty for precision on tasks that most applications do not need. What you can do is build systems that understand the limits of what the model can see, route character-level and arithmetic tasks appropriately, and never estimate context or cost from character counts when you have a tokenizer sitting right there.

The next article in this section, Embeddings and Positional Encoding: How Tokens Become Geometry, picks up exactly where tokenization ends: what happens to those integer token IDs once the model has them, and how they become the high-dimensional vectors that the transformer actually operates on.