Mixture of Experts (MoE), Explained

Modern AI models keep getting bigger, yet they often answer faster and cost less to run. That paradox is the cleanest way to grasp Mixture of Experts explained in plain terms: it is a design that lets a model grow enormous in size while staying light on every single query.

In this guide you will learn what an “expert” and a “router” actually are, why “total parameters” and “active parameters” are not the same number, and how real systems such as Switch Transformer, Mixtral, and DeepSeek V3 put the idea to work. We will also cover the price MoE pays in memory and engineering complexity. By the end, you should be able to answer a deceptively simple question on your own: why is a frontier-scale model often far cheaper to serve than its raw size suggests?

Mixture of Experts, Explained: A One-Sentence Definition

Mixture of Experts (MoE) is a neural-network design that breaks one large model into many smaller sub-networks called “experts,” and then activates only a handful of them for each piece of input. Instead of forcing one monolithic network to handle every token (a token is the small chunk of text — a word or word-piece — that a model processes), MoE routes each token to the experts best suited to it.

The core idea is “conditional computation.” The model keeps all of its capacity — every parameter stays in place — but only the parts relevant to the current input switch on. That lets engineers scale a model’s total size and its per-token compute cost independently, something a traditional dense network simply cannot do.

Why MoE? The Wall Dense Models Hit

A standard transformer is a “dense” model. Every time it processes a single token, it runs that token through all of its parameters, top to bottom. The smarter you make the model by piling on parameters, the heavier every token becomes to compute.

That creates a brutal cost curve. Roughly double the parameters and you roughly double the compute needed for both training and inference, so scaling turns expensive fast. This same dynamic sits at the heart of the broader AI compute cost debate.

Mixture of Experts attacks that curve directly. Keep growing the model’s capacity but switch on only a small slice of it per token, and you end up with a model that is large in knowledge yet cheap to run.

How MoE Works: Experts, Router, and Sparse Activation

The change happens inside the transformer block, at the feed-forward network (FFN, the small sub-network that transforms each token). A dense model places one large FFN at that position. An MoE layer replaces that single FFN with several smaller FFNs sitting side by side — and those are the experts.

In front of the experts sits a “router,” also called a gating network. It is usually a lightweight classifier — a small linear layer followed by a softmax — that scores how well each expert fits the current token. The router then forwards the token only to the top-scoring experts and leaves the rest switched off for that token.

How many experts fire depends on the model. Switch Transformer took the simplest path with top-1 routing, sending each token to a single expert, while Mixtral activates two of its eight experts (top-2). Selecting only a subset like this is called “sparse activation,” and it is where MoE’s efficiency begins. The number of experts you light up is ultimately a design choice that trades quality against cost.

Total vs Active Parameters: Where the Efficiency Lives

Two numbers unlock almost everything about MoE: total parameters and active parameters. Total parameters are all the weights a model owns, and they determine how much memory it needs to load. Active parameters are the ones actually used to process a single token, and they determine speed and compute cost (measured in FLOPs, or floating-point operations).

Mixtral 8x7B is the textbook case. Its total parameter count is 46.7B, but only 12.9B are active per token. That is exactly why Mistral describes it as running “at the same speed and cost as a 12.9B model.”

Put simply, total parameters tell you how much a model knows, while active parameters tell you how hard it works on each answer. The wider that gap, the bigger the efficiency MoE delivers.

Real MoE Models: Switch, Mixtral, and DeepSeek

It helps to ground the concept in real systems. Google’s Switch Transformer, published in 2021, simplified routing to top-1 and scaled models into the trillion-parameter range, reporting up to a 4x pre-training speedup over T5-XXL and as much as 7x over the smaller T5-Base and T5-Large models.

Mistral released Mixtral 8x7B in December 2023 under an Apache 2.0 license. It activates two of its eight experts per layer, beats Llama 2 70B on most benchmarks while running inference roughly 6x faster, and matches GPT-3.5-class quality (an MT-Bench score of 8.30).

DeepSeek V3, released in late 2024, pushed MoE further still. It carries 671B total parameters but activates just 37B per token, pairing Multi-head Latent Attention (MLA) with a DeepSeekMoE design. It trained on 14.8 trillion tokens using roughly 2.788M H800 GPU hours, and introduced an auxiliary-loss-free method for keeping its experts balanced.

Line the three up and a pattern emerges. The expert counts and activation ratios differ, but the skeleton is identical: a router lights up only some experts per token. Mixtral ships under Apache 2.0 and DeepSeek V3 arrived with a detailed technical report, so anyone can inspect how they are built. That openness is part of why MoE has moved from a research curiosity to a publicly vetted, standard design.

The Trade-offs: Memory, Load Balancing, and Complexity

MoE is not free. Even though only a few experts fire for any token, the model cannot know in advance which ones a future token will need, so every expert’s weights must stay resident in memory (VRAM, the working memory on a GPU). That is why Mixtral computes with just 12.9B parameters yet still demands the VRAM of a roughly 47B model.

The second challenge is balance. Left alone, a router tends to over-use a few popular experts while the rest sit idle — a failure mode known as “expert collapse.” Training therefore adds an auxiliary load-balancing loss to spread tokens more evenly, and DeepSeek V3 notably proposed a way to reach balance without that auxiliary loss at all.

The third cost is operational complexity. Routing, load balancing, and distributing experts across hardware all add moving parts, which makes MoE pipelines harder to design and stabilize than a dense model of equivalent quality.

A Short History: 1991 → 2017 → Today

MoE looks like a 2020s trend, but its roots run deep. The idea traces back to a 1991 paper, “Adaptive Mixture of Local Experts,” which already proposed training several expert networks alongside a gate that decides between them.

It entered modern deep learning in 2017, when Shazeer and colleagues applied a sparsely-gated MoE to a 137B-parameter LSTM and showed that activating only a slice of a giant network actually works at scale. Switch Transformer then settled the approach into a simple, scalable form for transformers in 2021, becoming the template that runs through Mixtral and DeepSeek today. This lineage also overlaps with the rise of reasoning models and the broader push for efficient AI.

In short, the history of MoE is the evolution of one question: how do you keep making a model bigger while spending less compute? Whenever a new large model launches, comparing its total parameters against its per-token active parameters tells you at a glance whether it leans on MoE and where its efficiency comes from.

Frequently Asked Questions (FAQ)

Q1. What is Mixture of Experts (MoE)?

It is an AI model design that splits one large neural network into many smaller “expert” sub-networks, with a router (gating network) activating only a few of them for each input token. Unlike a dense model that computes every parameter every time, MoE fires only the chosen experts, so capacity grows while per-token compute stays small. That trade is what people mean by “conditional computation.”

Q2. How is MoE different from a dense model?

A dense model runs every parameter for each token it processes. An MoE model computes only the small set of experts the router selects, so even at the same total size its per-token compute (active parameters) is far lower. The result is similar quality delivered faster and more cheaply.

Q3. What is the difference between active and total parameters?

Total parameters are all the weights a model holds, and they set the memory (VRAM) it needs. Active parameters are the ones actually computed for a single token, and they set speed and cost (FLOPs). Mixtral 8x7B, for example, has 46.7B total parameters but only 12.9B active per token.

Q4. Why do MoE models use so much memory?

Because even though only some experts activate for a given token, the model cannot predict which experts a future token will require, so all of them must stay loaded in memory. Mixtral therefore needs the VRAM of roughly a 47B model despite activating just 12.9B parameters.

Q5. How does the router (gating network) pick experts?

The router is usually a small linear layer plus a softmax that scores every expert for each token and selects the top-k. Switch Transformer uses top-1 (one expert) and Mixtral uses top-2 (two experts). An auxiliary load-balancing loss keeps tokens from piling onto a few favored experts.

Q6. What are the best-known MoE models?

Google’s trillion-scale Switch Transformer, Mistral’s Mixtral 8x7B (46.7B total, 12.9B active), and DeepSeek V3 (671B total, 37B active) are the headline examples. They all share the same principle of routing each token to only a few experts. Widely adopted in open source, MoE has become the standard way to run very large models at a reasonable cost.

References

• Hugging Face — Mixture of Experts Explained (2023-12-11) — MoE definition: a dense FFN is replaced by several experts plus a router, activating only a subset per token (conditional computation).
• Hugging Face — Mixture of Experts Explained (2023-12-11) — History: the 1991 Adaptive Mixture of Local Experts paper and Shazeer et al.’s 2017 sparsely-gated MoE on a 137B LSTM.
• Hugging Face — Mixture of Experts Explained (2023-12-11) — Total vs active parameters, and the requirement that all experts stay in VRAM (Mixtral ~47B-equivalent memory).
• Switch Transformers, Fedus et al., arXiv:2101.03961 (2021-01-11) — Top-1 routing, trillion-parameter scaling, and up to 4x (T5-XXL) / 7x (smaller T5) pre-training speedups.
• Mistral AI — Mixtral of experts (2023-12-11) — Mixtral 8x7B: 46.7B total / 12.9B active, two of eight experts per token, 32K context, Apache 2.0.
• Mistral AI — Mixtral of experts (2023-12-11) — Performance: beats Llama 2 70B on most benchmarks with ~6x faster inference, GPT-3.5-class (MT-Bench 8.30).
• DeepSeek-V3 Technical Report, arXiv:2412.19437 (2024-12-27) — 671B total / 37B active per token, using Multi-head Latent Attention (MLA) and DeepSeekMoE.
• DeepSeek-V3 Technical Report, arXiv:2412.19437 (2024-12-27) — Auxiliary-loss-free load balancing and multi-token prediction; 14.8T training tokens at ~2.788M H800 GPU hours.

jugger

Jugger | Backend Developer

A backend engineer working in enterprise solutions software since June 2021. The hands-on perspective from system design, API implementation, and production operations directly informs this site’s
AI, IT, and cloud infrastructure coverage.

For topics outside direct professional experience—blockchain and macroeconomics—analysis is built on primary-source citations such as SEC filings, whitepapers, institutional research, and government
statistics. Speculation is clearly separated from verified fact, and unverified claims are not published.

AI tools assist with drafting, but every article is reviewed, revised, and verified before publication. Errors are corrected with visible notes when discovered.

Contact: jjunggulee@gmail.com