#llm-internals
31 articles
Mixture of Experts: why every frontier model went sparse
How mixture of experts works — routers, top-k selection, shared experts — and the VRAM and batching math that decides when sparse models save you money.
Reading LLM Benchmarks Without Getting Fooled
LLM leaderboard scores are routinely inflated by contamination and benchmark saturation. Learn how to read them skeptically — and what to do instead.
RLHF, Reward Models, and Constitutional AI
How reinforcement learning from human feedback actually works: the reward model, PPO loop, RLAIF, and Constitutional AI — the baseline before DPO shortcuts.
Why Models Hallucinate: A Mechanistic View
Why LLMs hallucinate is not a mystery — it is a structural consequence of next-token prediction, probabilistic decoding, and a missing verification loop.
Reasoning Distillation: Teaching Small Models to Think
How DeepSeek-R1 distillation lets a 7B model out-reason a 32B baseline — the mechanics, the learnability gap, and what breaks in production.
How reasoning models are trained: RLVR, GRPO, and the post-training stack
The post-training recipe behind o1 and DeepSeek-R1: how RLVR and GRPO replaced RLHF for reasoning, with real training mechanics and failure modes.
Structured Outputs and Constrained Decoding: Guaranteeing Parseable JSON
How strict mode and grammar-constrained decoding actually guarantee parseable JSON — token masking, FSMs, xGrammar, and the validation layer you still need.
Decoding and Sampling: Temperature, Top-P, Min-P, and When to Use Each
How temperature, top-p, top-k, and min-p reshape the next-token probability distribution — and which setting to reach for in production vs creative tasks.
Capability limits and failure modes every AI engineer must know
Hallucination, lost-in-the-middle, injection, and benchmark overfitting are structural properties of LLMs — and they shape how you design systems.
Speculative decoding: 2-3x latency speedup without changing model quality
How a small draft model proposes tokens the large target verifies in parallel — and why rejection sampling guarantees identical output distributions with real speedup numbers.
Context Windows, KV Cache, and the Reality Behind 1M Token Claims
What context windows actually cost in GPU memory, why effective recall degrades before the advertised limit, and how to make smart architectural choices about long context.
Preference Optimization: DPO, SimPO, ORPO, and the Post-RLHF Stack
How DPO, SimPO, ORPO, and GRPO replaced the classic RLHF pipeline — what each method removes, when to use which, and the 2025 reasoning-alignment frontier.
The Transformer Block: Residual Stream, MLP, and What Each Part Learns
How attention and MLP sublayers share a residual stream, what factual knowledge each stores, and why this view changes how you debug and fine-tune LLMs.
Modalities Explained: From Text-Only to Omni Models
What modalities are, how frontier models encode images, audio, and video into transformer attention, and how to pick the right model for your pipeline in 2026.
LoRA, QLoRA, and the PEFT Landscape
How low-rank adapters let you fine-tune a 70B model on a single GPU, with the rank, alpha, and target-module settings that actually matter in production.
Test-Time Compute Scaling: Sequential, Parallel, and Tree-Based Strategies
How reasoning models spend more compute at inference time via longer chains, Best-of-N sampling, and tree search — and the cost, latency, and accuracy tradeoffs of each.
Few-shot prompting, chain-of-thought, and reasoning models
Master few-shot prompting and chain-of-thought, then learn why both techniques must be rethought entirely for o3, Claude extended thinking, and Gemini 2.x.
Self-Attention: The Mechanism That Reads Every Token Against Every Other
How self-attention computes dynamic, context-dependent relevance weights between all token pairs, why it scales quadratically, and what that costs you in production.
The model landscape in 2025-2026: closed frontier, open-weight, and reasoning
GPT-5, Claude Opus 4, Gemini 3.1, Llama 4, DeepSeek-R1: what the three competitive tiers actually mean for engineers picking models in production.
The KV Cache: GPU Memory Math Every Serving Engineer Must Know
A concrete formula for how much GPU memory your KV cache consumes, why it scales brutally with context length, and the PagedAttention trick that fixes it.
Embeddings and positional encoding: how tokens become geometry
How token embedding tables and positional encodings work, why RoPE dominates 2025 models, and what context-window advertising actually hides from you.
How Vision-Language Models Work: Architecture from CLIP to Native Multimodal
From CLIP contrastive pairs to natively multimodal transformers — how vision-language models actually see images, and what that means in production.
Token Economics: Where Your API Bill Actually Comes From
Dissect the full LLM API cost structure: input vs output token pricing, context bloat in multi-turn sessions, and the multiplier effect of long system prompts.
Prefill vs Decode: The Two Phases of Every LLM Request
Why the first token takes longer than every other token, how compute-bound prefill and bandwidth-bound decode shape your entire serving architecture, and what to do about it.
What Are Reasoning Models? Chain-of-Thought, o1, and the Inference-Time Shift
How o1-style models differ from standard LLMs: internal chain-of-thought via RL, not prompted CoT, and why that changes your prompting, cost model, and architecture.
Tokenization and BPE: Why your model cannot spell strawberry
How BPE tokenization works, why it makes English cheap and Arabic expensive, and why LLMs fail at counting letters and doing arithmetic.
What generative AI actually does: the next-token prediction loop
Strip away the hype: GenAI is autoregressive next-token prediction over a learned probability distribution. Understanding that loop explains both its power and its failure modes.
The Prediction Loop: What Generative AI Actually Does
GenAI is one loop repeated: predict the next token. Understanding that loop explains the power, the cost model, and every major failure mode — including why models can't count R's.
Transformers Without the Math
Build the mental model every AI engineer needs: how tokens become vectors, why attention is quadratic, what the MLP layers store, and how depth stacks understanding.
Tokens, Costs, and the Context Window
Why text becomes tokens, how input/output pricing asymmetry drives your bill, what the context window actually limits, and how prompt caching cuts costs by up to 90%.
Fine-Tuning: The Decision Framework
When to fine-tune, when to stay with RAG or prompting, and how LoRA, DPO, and distillation actually work — with the cost math to make the call.