#serving
10 articles
Mixture of Experts: why every frontier model went sparse
How mixture of experts works — routers, top-k selection, shared experts — and the VRAM and batching math that decides when sparse models save you money.
Deploying LLM Workloads: GPUs, Kubernetes, and Serverless
Where LLM serving engines actually run: GPU cloud choices, Kubernetes patterns, serverless GPU tradeoffs, cold starts, and autoscaling strategies that hold up under real traffic.
Throughput vs Latency: The GPU Economics of LLM Serving
How tokens/sec, cost per token, TTFT SLAs, and GPU utilization interact — and the engineering decisions that separate a profitable inference deployment from a money pit.
Choosing a serving engine: vLLM, SGLang, TensorRT-LLM, and Ollama in 2026
A decision framework for picking the right LLM inference engine based on workload, hardware, and ops complexity — grounded in 2025-2026 benchmarks.
Speculative decoding: 2-3x latency speedup without changing model quality
How a small draft model proposes tokens the large target verifies in parallel — and why rejection sampling guarantees identical output distributions with real speedup numbers.
Streaming, Batching, and Throughput: Latency vs Cost Trade-offs in Production
How streaming, continuous batching, and async batch APIs interact with latency and cost — and how to configure each without destroying p99 for interactive users.
Quantization Demystified: GPTQ, AWQ, GGUF, and FP8 With Real Tradeoffs
A practical comparison of GPTQ, AWQ, GGUF, and FP8 quantization formats — memory math, quality loss, hardware targets, and when each format actually wins in production.
Continuous Batching and Scheduling: Why Static Batching Wastes 70% of Your GPU
How iteration-level scheduling replaced static batching and became the single biggest serving optimization available without any model changes — with real numbers.
The KV Cache: GPU Memory Math Every Serving Engineer Must Know
A concrete formula for how much GPU memory your KV cache consumes, why it scales brutally with context length, and the PagedAttention trick that fixes it.
Prefill vs Decode: The Two Phases of Every LLM Request
Why the first token takes longer than every other token, how compute-bound prefill and bandwidth-bound decode shape your entire serving architecture, and what to do about it.