What Is Mixture of Experts (MoE)? Sparse Models, Explained

Mixture of Experts (MoE) is a neural network architecture that splits a model’s capacity across many specialized sub-networks — called experts — and activates only a small subset of them for any given input. The result is a model with a very large total parameter count but modest compute per token: the scale of a huge model at something closer to the inference cost of a small one. Nearly every frontier lab now ships MoE models, and the architecture is a big part of how modern large language models keep getting more capable without getting proportionally more expensive to run.

Dense models vs. sparse models

A standard dense model activates every parameter for every token it processes. Scale a dense model up and capacity and compute grow in lockstep — more capability, but every forward pass touches every weight, so training and serving costs climb with size.

A sparse model like MoE breaks that coupling. The total parameter count can be enormous, but only a fraction of those parameters do work on any given token, chosen dynamically based on what the input needs. Capacity scales; per-token compute doesn’t. That single idea is the core of MoE.

How routing works

Inside an MoE model, the feed-forward layers (where most parameters live in a transformer) are replaced with a set of expert feed-forward networks plus a lightweight router, also called a gating network. For each token, the router computes a score over all experts and selects the top-k — often two, sometimes eight of a much larger pool — to process that token. The selected experts run, and their outputs are combined as a weighted sum using the router’s scores.

The router is a small learned layer, trained jointly with the rest of the model — nobody hand-assigns topics to experts. In theory the router learns to specialize experts for different kinds of content: syntax, facts, code, multilingual text. In practice, researchers who inspect trained MoE models find the specialization is diffuse — experts often key on token-level patterns rather than clean human-readable subjects. The efficiency win doesn’t depend on interpretable specialization, though; it only requires that the router consistently pick experts that do the job well.

A short history: 1991 to the frontier

The idea is much older than the LLM era:

• 1991 — Jacobs, Jordan, Nowlan, and Hinton publish “Adaptive Mixtures of Local Experts,” the original formulation: multiple networks plus a gate that learns which one to trust for each input.
• 2017 — Noam Shazeer and colleagues at Google Brain publish the sparsely-gated MoE layer, embedding thousands of experts inside recurrent networks and scaling to over 100 billion parameters — the proof that sparse activation could deliver scale that dense training couldn’t. Shazeer went on to co-author the transformer paper itself, and his career since has tracked the architecture’s rise.
• 2020–2021 — Google’s GShard and Switch Transformer bring MoE to transformers at scale. Switch Transformer simplifies routing to top-1 (each token visits exactly one expert) and demonstrates trillion-parameter sparse models.
• December 2023 — Mistral AI releases Mixtral 8x7B with open weights: 8 experts per layer, top-2 routing, 46.7B total parameters but only about 12.9B active per token. It performs like a much larger dense model, and MoE stops being a proprietary trick.
• December 2024 — DeepSeek-V3 lands with 671B total parameters and roughly 37B active per token, using fine-grained experts (hundreds of small experts, several activated) plus a shared expert that every token visits. It becomes the template for a generation of open MoE models.
• Since then — the pattern is everywhere: Moonshot’s Kimi K2 pushed toward a trillion total parameters with ~32B active (see what Kimi is), and China’s open-weight leaders like the GLM family (see GLM-5.2) are MoE through and through.

Active vs. total parameters

Two numbers define every MoE model, and confusing them causes most MoE misunderstandings:

• Total parameters — the full size of the model, all experts included. This determines memory: every expert must be loaded and ready, because any token might need it.
• Active parameters — the parameters actually used per token. This determines compute per forward pass, which is what dominates inference cost.

Mixtral 8x7B is a 46.7B-parameter model that computes like a ~13B one. DeepSeek-V3 is a 671B-parameter model that computes like a ~37B one. You need hardware big enough to hold the whole thing — which is why quantization is almost always applied to MoE models to shrink their memory footprint — but each token’s forward pass is dramatically cheaper than a dense model of the same total size.

Load balancing: MoE’s hardest problem

Routing is learned, and learned systems find shortcuts. If a few experts happen to get slightly better early in training, the router sends them more tokens, they improve further, and the rest of the experts atrophy — a failure mode called expert collapse. The result is a model that paid the memory bill for hundreds of experts but effectively uses a handful.

The classic fix is an auxiliary load-balancing loss: an extra training penalty that nudges the router to spread tokens roughly evenly across experts. It works, but it fights the main training objective — the model is being asked to route tokens well and evenly at the same time, and those goals conflict. Later designs refined this with router z-losses for numerical stability, and DeepSeek-V3 demonstrated an auxiliary-loss-free strategy that adjusts per-expert bias terms instead of adding a competing loss, keeping load even without taxing model quality.

Fine-grained and shared experts

Early MoE transformers used a few big experts — Mixtral’s 8 per layer. The newer fine-grained approach slices capacity into many small experts (DeepSeek-V3 uses 256 routed experts per MoE layer, activating 8) so the router can compose more precise combinations per token. Alongside them sits a shared expert that every token passes through, which handles the common ground — basic syntax and semantics — so the routed experts can spend their capacity on genuinely specialized behavior. Most recent open MoE models follow some version of this recipe.

Memory and the parallelism problem

MoE’s efficiency has a price, and it’s paid in infrastructure:

Memory pressure. All experts must reside in accelerator memory even when idle. A 671B-parameter model needs enough GPU memory for 671B parameters, no matter that each token touches 37B.

Expert parallelism. At scale, experts are sharded across many GPUs, and each token’s hidden state must travel to whichever devices host its selected experts and back — an all-to-all communication pattern that can bottleneck inference if the interconnect is slow. Serving a big MoE model efficiently is as much a networking problem as a compute one.

Training instability. Discrete routing decisions make optimization noisier than dense training, which is why the load-balancing machinery above took years to mature.

Is MoE just an ensemble?

No — and the distinction matters. An ensemble runs several complete models on every input and averages their answers, multiplying compute per query. MoE is a single model whose layers contain alternative paths, and each token takes only a few of them — compute per query stays low by design. The experts are not standalone models: an individual expert is just a feed-forward block that only makes sense inside the network around it. You can’t extract “the code expert” from an MoE model and run it alone. Ensembles buy reliability by spending more compute; MoE buys capacity while holding compute roughly constant.

Why frontier labs converged on MoE

The economics won. Training compute is dominated by floating-point operations per token, and MoE buys more capacity per FLOP than dense scaling. At serving time, providers care about throughput per GPU-dollar, and a model that computes like a 37B model while knowing what a 671B model knows is a straightforwardly better deal. Several frontier proprietary models, including versions of GPT-4 and Gemini, have been widely reported to use MoE internally, and the open-weight field has made the architecture its default at the high end.

MoE also pairs naturally with model distillation: labs train a giant sparse teacher, then distill it into small dense models for their fast, cheap tiers — sparse capacity where quality is the priority, dense simplicity where latency and cost are.

The takeaway

Mixture of Experts breaks the coupling between model capacity and per-token compute: most parameters stay dormant on any forward pass, and a learned router decides which experts each token needs. The trade-offs are real — memory requirements stay high, load balancing is a genuine engineering discipline, and distributed serving gets complicated — but the economics of scaling made MoE the frontier’s default architecture. When you see a model advertising “1T total, 32B active,” you now know exactly what both numbers mean and why they’re different.