Mixture of Experts Explained: The Architecture Behind Every Frontier Model

In 2023, the most capable open-weight models were still often discussed as dense transformers. By early 2026, many public model reports and research papers were describing sparse Mixture of Experts systems with very large total parameter counts but much smaller active subsets per token. That shift tells you a lot about where large language model architecture is heading: not only toward bigger monoliths, but toward smarter routing.

Mixture of Experts (MoE) isn't new. The core idea dates back to 1991. But the last two years have turned it from a research curiosity into a prominent architecture for frontier and open-weight models. DeepSeek-V3, Qwen3, Mixtral, Llama 4, and Grok-1 are all publicly associated with some variant of sparse expert routing; other closed frontier systems are harder to verify. Understanding how MoE works is now table stakes for anyone following AI development.

The Core Idea: Divide and Specialize

A standard dense transformer runs every input token through every parameter in the network. A 70-billion-parameter model uses all 70 billion parameters for every single token, whether the input is a calculus problem or a grocery list.

MoE takes a different approach. Instead of one massive feed-forward network (FFN) at each layer, it uses multiple smaller sub-networks called "experts." A routing mechanism (sometimes called a gating network) decides which experts process each token. In sparse MoE designs, only the selected experts participate in a given forward pass.

The result: total model capacity can grow faster than per-token active compute. That does not mean inference is "free" or directly comparable across systems; routing strategy, expert size, batch shape, memory bandwidth, and hardware placement all determine the actual cost.

The concept first appeared in Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton's 1991 paper "Adaptive Mixtures of Local Experts," which proposed a system where specialized sub-networks each handle a subset of training cases. A gating network learns which expert should process which input. The fundamental architecture hasn't changed. The scale has.

How Routing Works (And Why It's the Hard Part)

The router is the most critical component. It takes each token's hidden representation and produces a probability distribution over all available experts. The token gets sent to the top-scoring expert(s), and their outputs are combined (usually weighted by the router's scores).

This sounds simple. It isn't.

Top-K Routing

Most production MoE models use top-k routing, where each token selects its k highest-scored experts. Mixtral 8x7B uses top-2 routing: every token goes to 2 out of 8 experts. DeepSeek-V3 uses top-8 out of 256 routed experts, plus one shared expert that processes every token.

Google's 2021 Switch Transformer simplified this further with top-1 routing (each token goes to exactly one expert), which reduced communication overhead but required careful load balancing to avoid expert collapse.

Expert Choice Routing

In 2022, Google Research flipped the approach. Instead of tokens choosing experts, experts choose tokens. Each expert has a fixed buffer capacity and selects its top-k preferred tokens from the batch. This guarantees perfect load balance by construction, and Google reported 2x faster training convergence in their 8B/64-expert model compared to standard top-1 and top-2 approaches.

The tradeoff: some tokens might get processed by zero experts (dropped) or many experts (over-processed), which creates unpredictable quality variance at inference time.

Hash Routing

The simplest approach assigns tokens to experts deterministically using a hash function. It maintains perfect load balance and adds zero learnable parameters to the router. But it also ignores token content entirely, so experts end up learning overlapping representations. In practice, hash routing consistently underperforms learned routing methods.

The Load Balancing Problem

Left unconstrained, routers tend to collapse. A few experts get selected repeatedly, others receive almost no tokens, and the model effectively becomes a dense network with wasted parameters. This is called routing collapse, and it's the single most common failure mode in MoE training.

The standard fix is an auxiliary loss that penalizes uneven expert utilization during training. But this creates its own problem: too large an auxiliary loss interferes with the primary training objective and degrades model quality; too small, and collapse happens anyway.

DeepSeek-V3 introduced an auxiliary-loss-free approach that adds a dynamic bias term to expert affinity scores. At each training step, the system monitors expert load and adjusts the bias upward for underloaded experts and downward for overloaded ones. No gradient interference with the main loss. This innovation is one reason DeepSeek-V3 achieved frontier performance at a reported training cost of approximately $5.6 million, a fraction of comparable models.

The Model Comparison: MoE in 2026

The table below shows commonly cited MoE model specifications released between 2023 and early 2026. Treat the active-parameter figures as reported model metadata, not a universal cost model.

GPT-4 has never been officially confirmed by OpenAI as an MoE system. Treat third-party speculation about its expert count or parameter count as rumor, not architecture evidence.

Several trends stand out in publicly reported systems. First, expert pools are becoming more fine-grained. Second, the shared expert pattern, where one expert processes every token to maintain baseline quality while routed experts specialize, is increasingly visible. Third, active parameter counts are dropping relative to total parameters in some reported model families. The exact ratios are model-specific and should not be generalized across the whole field.

Why MoE Wins on Economics

The case for MoE comes down to simple math. Training and inference costs scale primarily with active parameters (the FLOPs per token), not total parameters.

DeepSeek-V3's technical report describes a comparatively efficient training run relative to some dense-model baselines, but reported training costs are not directly comparable across labs. Hardware pricing, accounting method, data mix, and benchmark selection all affect the comparison.

At inference time, sparse MoE models process tokens using only their active expert subset, but the cost comparison to dense models is workload-dependent. A model with a small active subset can still be expensive to serve if expert parameters must be distributed across devices, routed frequently, or kept resident in high-bandwidth memory.

The catch is memory. All parameters (active or not) must be loaded into GPU memory, because the router needs access to every expert to make its selection. A 671B-parameter MoE model needs the same VRAM as a 671B dense model, even though it only uses 37B parameters per forward pass. This is the central tension of MoE deployment, and it's why cost efficiency in inference remains an active research area.

The Engineering Challenges Nobody Talks About

MoE models are harder to serve than dense models of equivalent quality. The architecture introduces several deployment headaches that don't show up in benchmark scores.

Communication Overhead

When experts are distributed across multiple GPUs (expert parallelism), tokens must be routed between devices. NVIDIA's measurements show that communication can dominate MoE layer processing time during decode in some hardware configurations. As the number of active experts increases and individual expert sizes shrink, the ratio of communication to useful compute can get worse.

DeepSeek addressed this through a co-design of algorithms, frameworks, and hardware that achieves near-complete computation-communication overlap during training. But this required custom engineering that isn't available off the shelf.

Expert Parallelism vs. Load Balance

Standard model parallelism splits a model's layers across GPUs. Expert parallelism places different experts on different GPUs. This creates a new problem: if token routing is uneven (and it always is in practice), some GPUs sit idle while others are overloaded. Critical bottlenecks like inter-GPU communication and workload imbalances caused by real-world token routing patterns remain partially unsolved, even in 2026.

The MoETuner system (2025) demonstrated that balanced expert placement combined with intelligent token routing can reduce inference latency, but achieving this requires runtime monitoring and dynamic rebalancing that adds operational complexity.

Memory Fragmentation

Each expert is a separate FFN with its own weight matrices. With 256 experts (DeepSeek-V3), that's 256 separate parameter blocks that must be managed in GPU memory. This fragments memory allocation and reduces the effectiveness of standard memory optimization techniques designed for dense, contiguous parameter tensors.

Fine-Grained vs. Coarse-Grained: The Design Spectrum

Early MoE models used a few large experts. Mixtral 8x7B has 8 experts, each roughly 7 billion parameters. This coarse-grained approach is simple to implement and reason about, but limits the router's ability to specialize.

The industry trend has shifted toward more fine-grained architectures in several visible model families. DBRX, DeepSeek-V3, Qwen3, and Llama 4 all illustrate different points on that design curve, but the right expert count depends on model size, data diversity, and serving constraints.

More experts means more possible combinations, which theoretically allows finer specialization. But it also means each individual expert has fewer parameters and less capacity. The benchmark results for these models suggest the tradeoff favors fine-grained designs, at least at current scales.

An emerging variation is the hybrid approach. Jamba (AI21 Labs, 2024) interleaves Transformer attention layers with Mamba state-space layers, and applies MoE only to certain layers. Vendor-reported speedups on long contexts point toward a future where MoE isn't just about expert routing within transformers, but about mixing entire architectural approaches.

What the Research Frontier Looks Like

The 2025-2026 literature reveals several active research directions that will shape the next generation of MoE models.

Routing stability under reinforcement learning. When MoE models undergo RLHF or GRPO post-training, routing distributions can drift significantly. This increases the variance of importance sampling weights and can destabilize optimization, particularly with multiple active experts per token. DeepSeek-V3 used its auxiliary-loss-free balancing to mitigate this during its reinforcement learning stage, but the problem remains open for general-purpose solutions.

ReLU routing. A 2025 ICLR paper proposed replacing the standard top-k + softmax routing with ReLU activation and adaptive L1 regularization (called ReMoE). This makes the routing function fully differentiable, which eliminates the discrete top-k selection that prevents gradient flow through the router. ReMoE reportedly outperforms traditional top-k routing across multiple model sizes and expert counts.

Optimal expert count. A December 2025 paper asked "How Many Experts Are Enough?" and found that optimal expert count depends on the relationship between model capacity and data diversity. More isn't always better. Beyond a certain point, additional experts fragment knowledge without improving specialization.

Distributed inference optimization. GRACE-MoE (2025) introduced grouping and replication with locality-aware routing to reduce communication overhead in distributed MoE inference. MegaScale-MoE (2025) tackled large-scale communication efficiency for production deployments. Both represent the kind of systems-level engineering required to make MoE practical beyond research settings.

These aren't incremental improvements. They address the fundamental bottlenecks (routing instability, deployment cost, scaling laws) that will determine whether MoE remains the dominant architecture or gets superseded by something else entirely.

Where MoE Falls Short

MoE isn't a free lunch. The architecture carries genuine limitations that its proponents tend to understate.

Expert collapse remains unsolved in the general case. Despite DeepSeek's auxiliary-loss-free approach, most MoE training runs still require careful hyperparameter tuning to prevent routing collapse. The router can, as Cerebras researchers put it, "single-handedly destroy an MoE model." Even with implementations that appear sound, mysterious training instabilities still occur.

Memory requirements are punishing. A 671B MoE model needs the same memory as a 671B dense model, even though it performs the FLOPs of a 37B model. For organizations that can't afford multi-GPU clusters, this erases the efficiency advantage. The cost of frontier inference is falling, but MoE's memory footprint keeps it out of reach for many deployment scenarios.

Interpretability is harder. Dense models are already opaque. MoE adds another layer of opacity: which experts activated, why, and whether different routing paths produce meaningfully different outputs. Early research into MoE interpretability suggests that experts don't cleanly specialize by topic or task the way the architecture's metaphor implies. The routing is messier than the diagrams suggest.

Throughput advantage shrinks at long context. MoE's efficiency advantage is most pronounced for short-context, high-batch inference. As context lengths grow (and inference-time compute scaling becomes more common), the memory bandwidth bottleneck of loading all expert parameters dominates, and the gap between MoE and dense models narrows.

The Default Architecture, For Now

Many prominent frontier and open-weight releases now use some form of sparse expert routing. The reasons are straightforward: MoE is one of the clearest ways to scale model capacity without forcing active compute to grow in lockstep. The economics are compelling, but they are not automatic.

But MoE in 2026 looks very different from the concept described in 1991, or even from the Switch Transformer of 2021. Fine-grained routing with large expert pools is more visible. Shared experts are one stabilizing pattern. Auxiliary-loss-free balancing is one active line of work on the collapse problem. Hybrid architectures are mixing MoE with state-space models and other non-transformer components.

The architecture still has real problems: memory overhead, communication bottlenecks, routing instability, and interpretability gaps. And the research community hasn't converged on answers for any of them. Whether MoE stays dominant depends less on the quality of the routing algorithms and more on whether the engineering community can solve the systems-level deployment challenges at scale.

For now, if you're building on, evaluating, or competing with frontier models, MoE is part of the architecture landscape you need to understand. Whether it is the right fit depends on the serving constraints as much as the benchmark chart.

Key Takeaways

• Sparse activation lowers active compute, not total system complexity. Only selected experts process each token, but routing, memory residency, and communication overhead determine whether the serving economics work.
• Expert routing is the core engineering challenge. Load balancing, expert collapse, and unstable routing can erase the theoretical efficiency gains if they are not monitored and corrected.
• Reported active-parameter counts are not a complete cost model. They help explain why MoE differs from dense models, but scaling laws and optimization strategies do not transfer cleanly.
• MoE deployment is constrained by memory and interconnects. The full parameter set still has to be stored and served, which keeps large MoE systems in the domain of well-resourced infrastructure teams.

Related: AI Agents in Legal: What Works, What Fails,

Sources

Research Papers:

• Adaptive Mixtures of Local Experts — Jacobs, Jordan, Nowlan, Hinton (1991)
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Shazeer et al. (2017)
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Lepikhin et al. (2020)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Fedus, Zoph, Shazeer (2022)
• Mixture-of-Experts with Expert Choice Routing — Zhou et al. (2022)
• Mixtral of Experts — Jiang et al. (2024)
• DeepSeek-V3 Technical Report — DeepSeek-AI (2024)
• Qwen3 Technical Report — Qwen Team (2025)
• How Many Experts Are Enough? Towards Optimal Semantic Specialization for MoE — (2025)
• A Comprehensive Survey of Mixture-of-Experts — (2025)
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts — (2024)

Industry / Case Studies:

• Mixture of Experts Explained — Hugging Face
• Introducing DBRX: A New State-of-the-Art Open LLM — Databricks
• Router Wars: Which MoE Routing Strategy Actually Works — Cerebras
• MoE at Scale: Making Sparse Models Fast on Real Hardware — Cerebras
• MoE vs Dense Models: How Do They Compare in Inference? — Epoch AI
• Scaling Large MoE Models with Wide Expert Parallelism on NVL72 — NVIDIA
• The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation — Meta AI

Related Swarm Signal Coverage:

• DeepSeek Explained

• Inference-Time Scaling

• The Benchmark Crisis: Why Model Leaderboards Are Becoming Marketing Tools

• The Budget Problem

• The Frontier Model Wars: Gemini 3 vs GPT-5 vs Claude 4.5

• Qwen and the Open-Source Revolution

• MoE vs Dense Models: A Practitioner's Decision Guide for 2026