Sparsity

Plain English. In a sparse (mixture-of-experts, MoE) model, only a fraction of the weights fire for any given token — the model routes each token to a few specialist "experts" out of hundreds. The sparsity factor is the ratio of total to active parameters: a 552B-parameter model activating 8–16B per token is doing frontier work at small-model running cost.

Why it moves money. Sparsity is the main trick behind cheap frontier-adjacent models, and it decouples the two numbers investors habitually conflate: parameter count (capability headline) and active parameters (serving cost). It also changes what hardware matters — sparse models are why a 744B model can run from a consumer machine by streaming experts off disk.

What to watch. Active-parameter counts in model cards, not headline sizes. Serving economics follow the active number; marketing follows the total.

From the signals. DeepSeek V4.1-Flash: 552B MoE activating 8B per token in prefill, 16B in decode. DeepSeek V4 delivered frontier-adjacent capability at a tenth of the memory. A 744-billion-parameter model now runs on 25GB of consumer RAM by streaming its experts.

Further reading. Fedus, Zoph & Shazeer, "Switch Transformers" (2021).