Mixture of Experts: why every frontier model went sparse

The plan looked fine on the whiteboard. DeepSeek-V3 is open-weight, benchmarks near the frontier, and activates only 37B parameters per token — so it should serve roughly like a 37B model, and a two-GPU node should handle it. Then someone runs the memory math. All 671 billion parameters have to sit in VRAM whether or not they fire, which at FP8 is ~671 GB of weights before a single KV-cache byte. An entire 8×H100 node — 640 GB — cannot load the model. The revised quote is 8×H200 at roughly $30/hour rented. And when it finally boots, your actual traffic of a few concurrent users decodes at a few dozen tokens per second while eight of the most expensive GPUs money can rent sit ~95% idle. The API you were trying to replace charges about a dollar per million output tokens. Your rig is charging you two hundred.

Nothing malfunctioned. That is a mixture of experts (MoE) model doing exactly what it was designed to do, evaluated with dense-model intuitions. Since every open-weight frontier model of 2025-2026 is MoE — DeepSeek-V3 and R1, Qwen3's MoE line, Kimi K2, Llama 4, gpt-oss, the Mixtral lineage before them — those intuitions now fail by default. This article rebuilds them: first the mechanism, then the numbers that "active parameters" does and does not predict, then the batch-size economics that decide whether sparsity saves you money or sets it on fire.

What mixture of experts actually replaces

Look at where the parameters live in a transformer. In a standard transformer block, roughly two-thirds of the weights sit in the FFN — the two big matrices that expand the hidden state 4× and project it back. Attention is where the interesting mixing happens, but the FFN is where the bulk of the knowledge is stored, and every token pays for all of it on every layer.

MoE keeps the attention exactly as it is and attacks the FFN. Each MoE layer replaces the single dense FFN with N smaller expert FFNs — same shape, independent weights — plus a router: a small linear layer that maps the token's hidden state to a score per expert. Softmax the scores, keep the top-k, run only those experts, and output the score-weighted sum. Everything else in the block — attention, layer norms, residual stream — runs densely for every token, which is why "8×7B" Mixtral is 47B total rather than 56B: only the FFNs are replicated.

flowchart TD
    X["token hidden state x"] --> ATT["attention block — dense, always runs"]
    ATT --> RN["router: scores = softmax(W_r · x)"]
    RN -->|"keep top-8 scores"| SEL["8 routed experts run"]
    RN -.->|"248 experts idle for this token"| IDLE["idle experts — still resident in VRAM"]
    ATT --> SHARE["shared expert — every token"]
    SEL --> MIX["output = weighted sum + shared(x)"]
    SHARE --> MIX
    MIX --> NEXT["residual stream, next layer"]

    style RN fill:#00e5ff,color:#000
    style SEL fill:#0e7490,color:#fff
    style SHARE fill:#22c55e,color:#000
    style IDLE fill:#374151,color:#fff
    style MIX fill:#1e3a5f,color:#fff

Two design axes matter in practice. The first is granularity. Mixtral (December 2023) used 8 large experts and routed each token to 2 of them. DeepSeek-V3 went fine-grained: 256 small experts per layer, top-8 routing, and it keeps its first three layers dense. Fine-grained experts give the router 256-choose-8 combinations to express per token instead of 8-choose-2 — enormously more specialization for the same active budget. Everyone followed: Qwen3-235B-A22B routes 8 of 128, Kimi K2 routes 8 of 384, gpt-oss-120b routes 4 of 128.

The second axis is the shared expert: one FFN that every token passes through no matter what the router says. The logic — laid out well in Sebastian Raschka's MoE walkthrough — is that common patterns shouldn't be relearned redundantly in dozens of routed experts; park them in one always-on expert and let the routed ones specialize. DeepSeek-V3 and Kimi K2 use one shared expert; Qwen3 and gpt-oss skip it. Both camps ship frontier models, so treat it as a design choice, not doctrine.

The router itself is almost embarrassingly small — a single matrix of shape hidden_dim × num_experts:

def moe_layer(x, experts, shared, W_r, k=8):
    scores = softmax(x @ W_r)              # [num_experts] — the "router"
    topk_idx = argtop(scores, k)           # e.g. 8 of 256
    out = sum(scores[i] * experts[i](x)    # only k experts execute
              for i in topk_idx)
    return out + shared(x)                 # shared expert: always on

A few thousand parameters per layer decide which billions get used. Which raises the obvious question: what stops those few thousand parameters from making degenerate choices?

Keeping the router honest

Left alone, routers collapse. Early in training, whichever experts happen to receive slightly more tokens improve slightly faster, so the router prefers them more, so they receive even more tokens. The fixed point is a model that routes almost everything to a handful of experts while the rest stay near their random initialization — you paid for 256 experts and trained 12. The failure is old and well documented; the Switch Transformer work (Google, 2021), which scaled top-1-routed MoE past a trillion parameters, made the standard countermeasure famous: an auxiliary load-balancing loss that penalizes the router when the fraction of tokens per expert deviates from uniform, plus an expert capacity cap that drops tokens routed to an overfull expert.

The auxiliary loss works but it taxes quality — you are spending gradient signal on fairness instead of on modeling language. DeepSeek-V3's contribution here is an auxiliary-loss-free balancing scheme: each expert carries a bias added to its routing score only for top-k selection (not for the output weighting), and the bias is nudged down when the expert is overloaded and up when it is underused. Balance becomes a control loop instead of a loss term. And it worked at scale: the V3 report describes a conspicuously smooth run — 14.8T tokens with no irrecoverable loss spikes or rollbacks, 2.788M H800 GPU-hours total, about $5.6M at their assumed $2/GPU-hour rate. That last number is why every lab's 2025 roadmap suddenly had an MoE on it.

Balance matters at inference too, not just training. Expert usage is only uniform on average over the training distribution. Your traffic is not the training distribution — if it is 90% Python, the experts that Python routes to become hot spots while the rest of the machine idles. Hold that thought; it comes back in the serving section.

What "active parameters" does and does not predict

Here is the table to keep in your head, with the rule-of-thumb column doing the useful work (specs as of mid-2026):

Model	Total	Active	Experts per MoE layer	Routing	Shared expert	~√(active·total)
Mixtral 8x7B	47B	13B	8	top-2	no	~25B
gpt-oss-120b	117B	5.1B	128	top-4	no	~24B
Qwen3-235B-A22B	235B	22B	128	top-8	no	~72B
Llama 4 Maverick	400B	17B	128	top-1 + shared	yes	~82B
DeepSeek-V3 / R1	671B	37B	256	top-8	yes	~158B
Kimi K2	~1T	32B	384	top-8	yes	~179B

Active parameters predict per-token compute. A decoded token costs roughly 2 FLOPs per active parameter: ~74 GFLOPs for DeepSeek-V3 against ~1.3 TFLOPs for a hypothetical dense 671B. That is the 18× that makes the architecture worth the trouble.

Total parameters predict memory. Every expert must be resident because the router's choice changes token to token, layer to layer. There is no useful sense in which you can load "just the active 37B."

Neither predicts capability. The rule of thumb that circulates among practitioners — and that holds up reasonably well against the models above — is that an MoE performs like a dense model at the geometric mean of active and total parameters. DeepSeek-V3 behaves like a ~160B dense model: far beyond any 37B, nowhere near what a dense 671B trained on the same data would be. gpt-oss-120b lands in ~24B dense territory, which matches how it actually feels to use. It is a heuristic, not a law — data quality and architecture shift it — but it will keep you from being fooled in either direction by a spec sheet. Epoch AI's analysis makes the sharper point that even the cost-equivalent dense size isn't one number: it depends on serving regime, because prefill is compute-bound and fast decoding is bandwidth-bound, and sparsity helps far more in one than the other. Which brings us to the part that determines your bill.

The economics: buying memory to save FLOPs

Strip away the architecture and MoE is one trade: a fixed cost (VRAM to hold every expert, billed hourly whether or not tokens flow) in exchange for a lower marginal cost (FLOPs per token). Any trade of that shape has a break-even volume. Below it you lose; above it you win big. Everything confusing about MoE serving economics is this one sentence wearing different hats.

Run the numbers honestly:

DeepSeek-V3, FP8 weights (1 byte/param) — illustrative mid-2026 rates

Memory (fixed cost):
  weights: 671B × 1 B                      ≈ 671 GB
  8× H100 (80 GB)  = 640 GB                → does not fit. At all.
  8× H200 (141 GB) = 1,128 GB              → fits, ~450 GB left for KV cache
  rental: 8× H200                          ≈ $30/hour, tokens or no tokens

Compute per decoded token (marginal cost):
  MoE:        2 × 37B active               ≈ 74 GFLOPs
  dense 671B: 2 × 671B                     ≈ 1,342 GFLOPs   (18× more math)

Single stream (batch = 1):
  weights read per token ≈ 37 GB (top-8 + shared, FP8)
  roofline: 38.4 TB/s aggregate HBM ÷ 37 GB ≈ 1,000 tok/s in theory
  reality: ~20-60 tok/s — two all-to-all hops × 58 MoE layers,
  attention, kernel launches. Latency, not bandwidth, binds at batch 1.

Cost per token at batch = 1:
  $30/hr ÷ (40 tok/s × 3,600 s)            ≈ $208 per 1M output tokens
  DeepSeek's API (same weights, saturated): ~$1 per 1M output tokens
  → self-hosting at low utilization: ~100-200× worse. Per token.

Saturated (hundreds of concurrent sequences):
  every expert busy every step; weight traffic ≈ all 671 GB per step,
  amortized over the batch → per-token traffic collapses toward
  active-param levels, throughput reaches thousands of tok/s per node,
  and cost/M tokens falls ~100× — into the range the API charges.

The API is not cheap because of some subsidy. The API is cheap because it is saturated — your request shares each forward pass's weight reads with hundreds of strangers' requests. Batching is the entire business model, and continuous batching is the machinery that keeps the pool full.

{ "type": "batching", "mode": "continuous", "title": "Why saturated batches are the whole MoE business model" }

There is a subtlety that makes MoE's break-even volume higher than a dense model's, and it is worth understanding because it is where dense intuition fails hardest. Decode throughput is governed by how many bytes of weights you stream from HBM per generated token — the prefill/decode split exists because prefill is compute-bound and decode is bandwidth-bound. A dense model reads all its weights every step at any batch size, so batching amortizes weight reads linearly from batch 1. An MoE at batch 1 reads only the active experts — but each additional sequence in the batch routes to different experts. With top-8-of-256 routing, a batch of just 32 tokens already touches ~160 of the 256 experts per layer in expectation. Long before your batch is big enough to amortize anything, you are streaming nearly all 671 GB per step. In the awkward middle — batch 8 to 64, exactly where most self-hosted deployments live — you pay dense-671B memory traffic for 37B-worth of useful compute per token. The curve does not bend in your favor until the batch is large enough that hundreds of tokens share each expert's weight read, and on an H100-class part the compute-bound crossover sits around 300 tokens per forward pass. Dense models forgive mediocre utilization. MoE punishes it.

So, the answer the title promised: labs went sparse because both of their constraints favor it. Training-side, MoE reaches better loss per training FLOP — DeepSeek-V3's $5.6M-scale run is the receipt. Serving-side, labs run saturated fleets where the fixed memory cost is amortized to nothing. Your constraints are probably neither. If your traffic is ten concurrent conversations, the architecture that a frontier lab chose for its economics is close to the worst thing you could self-host — a dense model on one right-sized GPU beats it on cost and on time-to-first-token, since the throughput-vs-latency trade at low batch is dominated by how much hardware you had to rent to fit the weights at all.

Serving a big MoE: expert parallelism and the all-to-all tax

A model that doesn't fit on one GPU has to be sliced, and MoE gets its own slicing strategy. Tensor parallelism — splitting every matrix across GPUs — works but wastes the structure. Expert parallelism (EP) uses it: shard the experts across GPUs (256 experts on 8 GPUs = 32 each), replicate or shard everything else, and move tokens to experts instead of splitting every matrix multiply.

The price is communication. For every MoE layer, each GPU must send each of its tokens' hidden states to whichever GPUs hold that token's selected experts, then gather the outputs back: an all-to-all dispatch and an all-to-all combine, per layer, per step.

sequenceDiagram
    participant G0 as GPU 0 (experts 0-31)
    participant G1 as GPU 1 (experts 32-63)
    participant G7 as GPU 7 (experts 224-255)

    Note over G0,G7: MoE layer N — each GPU holds a slice of the experts
    G0->>G0: router scores its tokens, picks top-8 experts each
    G0->>G1: dispatch hidden states routed to experts 32-63 (all-to-all)
    G0->>G7: dispatch hidden states routed to experts 224-255
    G1->>G1: expert FFNs run on received tokens
    G7->>G7: expert FFNs run on received tokens
    G1-->>G0: combine expert outputs back (all-to-all)
    G7-->>G0: combine outputs back
    Note over G0,G7: two network hops per MoE layer — 58 layers, every token, every step

For DeepSeek-V3 that is ~116 all-to-all operations per token per forward pass. This is why MoE serving is an interconnect problem before it is a FLOPs problem: NVLink-class bandwidth inside the node is what makes the architecture viable, and any straggler GPU stalls the whole layer, because the combine cannot finish until the slowest expert does. Remember the inference-time balance problem from the training section? This is where it bites — a hot expert (your all-Python traffic) turns into a hot GPU, and a hot GPU turns into a tail-latency floor for every request in the batch. Production stacks counter with redundant replicas of hot experts and dynamic expert placement; DeepSeek's own serving system and the large-scale EP modes in vLLM and SGLang (see the serving-engine comparison) exist substantially to fight this one fire. At real scale, deployments go wider than one node — "wide EP" across tens of GPUs with prefill and decode disaggregated — at which point you are operating a distributed system whose failure domain is the model.

Two mitigations change the picture at the small end. Quantization attacks the fixed cost directly: at 4-bit, DeepSeek-V3 is ~340 GB — still multi-GPU, but half the hardware — and the quantization playbook applies to experts as well as it does to dense weights. And small MoEs dodge the tax entirely: gpt-oss-120b ships MXFP4-quantized to fit a single 80 GB GPU, and Qwen3-30B-A3B squeezes onto a 24 GB consumer card at 4-bit. One GPU means no all-to-all, no EP, no straggler problem — you keep the FLOPs savings and skip the distributed-systems bill. As of mid-2026, single-GPU MoE is the quiet sweet spot of the whole architecture.

One more serving note: MoE does nothing about the KV cache. Attention is dense, so KV-cache memory math applies unchanged, and at the batch sizes MoE needs to be economical the cache is enormous. DeepSeek attacks this with multi-head latent attention (MLA), which compresses each token's KV state into a small latent vector — roughly an order of magnitude below a conventional cache — less a bonus feature than a load-bearing one, since without it the saturated batches that justify the MoE wouldn't fit next to the weights.

What breaks

You sized hardware off active parameters. The classic. "37B active" goes into a capacity plan, procurement orders two H100s, the model needs eight H200s. Nothing downstream of that spreadsheet error is recoverable. VRAM = total params × bytes per param, plus KV cache, always.

You hit the utilization cliff. The deployment works in the load test — which ran saturated — and then production traffic turns out to be 5-30 concurrent sequences with bursty peaks. Cost per token comes in 20-100× over the estimate, and the postmortem discovers the estimate silently assumed the batch sizes from the benchmark. This is the single most common way self-hosting a big MoE disappoints against the API, and no config change fixes it: it is the architecture doing what it does at the volume you actually have.

Hot experts create stragglers. Uniform expert load is a training-distribution average. Skewed production traffic — one language, one domain, one template stamped across every request — concentrates routing onto a few experts, and under expert parallelism those become hot GPUs that gate every layer's combine step. Symptoms: aggregate throughput well below benchmark at the same batch size, p99 latency drifting with traffic mix. If your serving stack supports expert-load metrics and redundant expert replicas, this is what they are for.

CPU offloading "works." llama.cpp and ktransformers will happily run a 4-bit DeepSeek-V3 with most experts in system RAM, streaming them over PCIe on demand. It runs. It runs at a few tokens per second with time-to-first-token measured in geologic time. Fine for evaluating the model overnight; a trap the moment anyone says "we could serve this."

Fine-tuning is harder than the blog post implied. Full fine-tuning means every expert's weights plus optimizer state resident across a training cluster — for V3-class models, out of reach for almost everyone. LoRA-style adaptation works but the tooling is dense-first: many recipes adapt only attention and shared paths, and if the router shifts under distribution-skewed fine-tuning data you can unbalance expert load in ways that surface later as serving pathologies. Budget more experimentation time than you would for an equivalent dense fine-tune, or fine-tune a small dense model instead and keep the MoE frozen behind it.

The decision in practice

flowchart LR
    Q["your workload"] --> B{"can you keep batches\nsaturated all day?"}
    B -->|"yes — hundreds of\nconcurrent sequences"| MOE["self-host the big MoE\nwith expert parallelism"]
    B -->|"no — bursty or\nlow concurrency"| C{"can data\nleave your infra?"}
    C -->|"yes"| API["use the API — pay for\nsomeone else's saturation"]
    C -->|"no"| DENSE["dense 8-70B or single-GPU MoE\non right-sized hardware"]

    style B fill:#ffaa00,color:#000
    style C fill:#ffaa00,color:#000
    style MOE fill:#22c55e,color:#000
    style API fill:#0e7490,color:#fff
    style DENSE fill:#0e7490,color:#fff

Use the API for frontier-scale MoE unless you can prove — with your real traffic trace, not the load test — sustained saturation. At saturation, self-hosting DeepSeek-V3-class models is genuinely cheap per token; that is not marketing, it is the same amortization the API providers enjoy, and if you have that volume plus a team that can operate wide expert parallelism, take it. The open-vs-closed decision framework covers the surrounding considerations, but for MoE specifically, utilization dominates everything else in the tradeoff.

If you must self-host below saturation — a compliance mandate, an air-gapped network — do not reflexively reach for the biggest open model. A dense 70B, or better, a single-GPU MoE like gpt-oss-120b, delivers most of the quality at a tenth of the fixed cost and none of the all-to-all operational surface. The geometric-mean rule is your ally here: a ~24B-equivalent model on one GPU you keep busy beats a ~160B-equivalent model on eight GPUs you don't.

And when you read the next model announcement, translate the headline before reacting: total parameters tell you the VRAM bill, active parameters tell you the FLOPs, the geometric mean tells you roughly what it can do, and none of those numbers tells you whether your traffic can keep it fed. That last one is the only question the spec sheet can't answer — and it is the one the bill depends on.