#tokenization
6 articles
Token Economics: Where Your API Bill Actually Comes From
Dissect the full LLM API cost structure: input vs output token pricing, context bloat in multi-turn sessions, and the multiplier effect of long system prompts.
Tokenization and BPE: Why your model cannot spell strawberry
How BPE tokenization works, why it makes English cheap and Arabic expensive, and why LLMs fail at counting letters and doing arithmetic.
What generative AI actually does: the next-token prediction loop
Strip away the hype: GenAI is autoregressive next-token prediction over a learned probability distribution. Understanding that loop explains both its power and its failure modes.
The Prediction Loop: What Generative AI Actually Does
GenAI is one loop repeated: predict the next token. Understanding that loop explains the power, the cost model, and every major failure mode — including why models can't count R's.
Transformers Without the Math
Build the mental model every AI engineer needs: how tokens become vectors, why attention is quadratic, what the MLP layers store, and how depth stacks understanding.
Tokens, Costs, and the Context Window
Why text becomes tokens, how input/output pricing asymmetry drives your bill, what the context window actually limits, and how prompt caching cuts costs by up to 90%.