#inference
29 articles
Mixture of Experts: why every frontier model went sparse
How mixture of experts works — routers, top-k selection, shared experts — and the VRAM and batching math that decides when sparse models save you money.
Deploying LLM Workloads: GPUs, Kubernetes, and Serverless
Where LLM serving engines actually run: GPU cloud choices, Kubernetes patterns, serverless GPU tradeoffs, cold starts, and autoscaling strategies that hold up under real traffic.
Build vs Buy and Small vs Large: The Framework for Model and Infrastructure Decisions
Self-hosting beats APIs only above ~$4,200/month. Small models win on latency and cost. Learn the decision framework that prevents both premature optimization and missed savings.
Throughput vs Latency: The GPU Economics of LLM Serving
How tokens/sec, cost per token, TTFT SLAs, and GPU utilization interact — and the engineering decisions that separate a profitable inference deployment from a money pit.
Limits and failure modes: overthinking, underthinking, and when reasoning breaks
The empirically-documented ways reasoning models fail in production — overthinking, underthinking, sycophantic traces, benchmark gap — and how to detect and route around them.
Why Models Hallucinate: A Mechanistic View
Why LLMs hallucinate is not a mystery — it is a structural consequence of next-token prediction, probabilistic decoding, and a missing verification loop.
Choosing a serving engine: vLLM, SGLang, TensorRT-LLM, and Ollama in 2026
A decision framework for picking the right LLM inference engine based on workload, hardware, and ops complexity — grounded in 2025-2026 benchmarks.
Structured Outputs and Constrained Decoding: Guaranteeing Parseable JSON
How strict mode and grammar-constrained decoding actually guarantee parseable JSON — token masking, FSMs, xGrammar, and the validation layer you still need.
Decoding and Sampling: Temperature, Top-P, Min-P, and When to Use Each
How temperature, top-p, top-k, and min-p reshape the next-token probability distribution — and which setting to reach for in production vs creative tasks.
Speculative decoding: 2-3x latency speedup without changing model quality
How a small draft model proposes tokens the large target verifies in parallel — and why rejection sampling guarantees identical output distributions with real speedup numbers.
Context Windows, KV Cache, and the Reality Behind 1M Token Claims
What context windows actually cost in GPU memory, why effective recall degrades before the advertised limit, and how to make smart architectural choices about long context.
Open vs. closed models: a decision framework, not a religion
A practical framework for choosing between open-weight and closed frontier models — covering cost math, data governance, performance gaps, and operational burden as of mid-2026.
Streaming, Batching, and Throughput: Latency vs Cost Trade-offs in Production
How streaming, continuous batching, and async batch APIs interact with latency and cost — and how to configure each without destroying p99 for interactive users.
Quantization Demystified: GPTQ, AWQ, GGUF, and FP8 With Real Tradeoffs
A practical comparison of GPTQ, AWQ, GGUF, and FP8 quantization formats — memory math, quality loss, hardware targets, and when each format actually wins in production.
Cost, latency, and the model-selection decision tree for reasoning
A practical framework for choosing between reasoning and standard models, with real cost and latency numbers from 2025-2026 deployments.
Continuous Batching and Scheduling: Why Static Batching Wastes 70% of Your GPU
How iteration-level scheduling replaced static batching and became the single biggest serving optimization available without any model changes — with real numbers.
Test-Time Compute Scaling: Sequential, Parallel, and Tree-Based Strategies
How reasoning models spend more compute at inference time via longer chains, Best-of-N sampling, and tree search — and the cost, latency, and accuracy tradeoffs of each.
Self-Attention: The Mechanism That Reads Every Token Against Every Other
How self-attention computes dynamic, context-dependent relevance weights between all token pairs, why it scales quadratically, and what that costs you in production.
The model landscape in 2025-2026: closed frontier, open-weight, and reasoning
GPT-5, Claude Opus 4, Gemini 3.1, Llama 4, DeepSeek-R1: what the three competitive tiers actually mean for engineers picking models in production.
The KV Cache: GPU Memory Math Every Serving Engineer Must Know
A concrete formula for how much GPU memory your KV cache consumes, why it scales brutally with context length, and the PagedAttention trick that fixes it.
How Vision-Language Models Work: Architecture from CLIP to Native Multimodal
From CLIP contrastive pairs to natively multimodal transformers — how vision-language models actually see images, and what that means in production.
Token Economics: Where Your API Bill Actually Comes From
Dissect the full LLM API cost structure: input vs output token pricing, context bloat in multi-turn sessions, and the multiplier effect of long system prompts.
Prefill vs Decode: The Two Phases of Every LLM Request
Why the first token takes longer than every other token, how compute-bound prefill and bandwidth-bound decode shape your entire serving architecture, and what to do about it.
What Are Reasoning Models? Chain-of-Thought, o1, and the Inference-Time Shift
How o1-style models differ from standard LLMs: internal chain-of-thought via RL, not prompted CoT, and why that changes your prompting, cost model, and architecture.
Tokenization and BPE: Why your model cannot spell strawberry
How BPE tokenization works, why it makes English cheap and Arabic expensive, and why LLMs fail at counting letters and doing arithmetic.
What generative AI actually does: the next-token prediction loop
Strip away the hype: GenAI is autoregressive next-token prediction over a learned probability distribution. Understanding that loop explains both its power and its failure modes.
The Prediction Loop: What Generative AI Actually Does
GenAI is one loop repeated: predict the next token. Understanding that loop explains the power, the cost model, and every major failure mode — including why models can't count R's.
The Model Landscape: Choosing Without Getting Played
A decision framework for picking between closed frontier models, open-weight, and reasoning tiers — plus benchmark literacy so leaderboard scores stop misleading you.
Reasoning Models: When and How to Use Them
Reasoning models spend inference tokens thinking before they answer. Learn when that's worth it, when it's a waste, how prompting changes, and how to control cost.