Paper digest: Mixture-of-Experts
Mixture-of-Experts lets AI models grow much larger without costing much more to run. How it works, from Google's early papers to Mistral's Mixtral.
Many of today's most capable AI models don't use their whole "brain" for every word. They use a technique called Mixture-of-Experts, which sends each word to a few specialist parts of the model and leaves the rest idle.
That lets companies build much bigger models without a matching rise in running costs. For anyone buying AI, it also explains why a model's advertised size doesn't tell you what it costs to use.
The papers
| Paper | Authors | Year | Link |
|---|---|---|---|
| Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Shazeer, Hinton, Dean and others (Google) | 2017 | arXiv:1701.06538 |
| GShard | Lepikhin and others (Google) | 2020 | arXiv:2006.16668 |
| Switch Transformers | Fedus, Zoph and Shazeer (Google) | 2021 | arXiv:2101.03961 |
| GLaM | Du and others (Google) | 2021 | arXiv:2112.06905 |
| Mixtral of Experts | Jiang and others (Mistral AI) | 2024 | arXiv:2401.04088 |
Why these papers exist
AI language models are built from repeated blocks, as explained in our Attention Is All You Need digest. In a standard model, every part of every block works on every word. Make the model twice as big and each word costs roughly twice as much to process.
Researchers wanted bigger models, because bigger models learn more, without that cost rising in step. Mixture-of-Experts was the answer.
How it works
Each block in a Transformer has a section that processes words one at a time. In a Mixture-of-Experts model, that section is replaced by several copies, called experts. A small "router" looks at each word and decides which one or two experts should handle it.
So the model can hold many experts in total, but each word only uses a few. The other parts of the block, which compare words with each other, stay shared.
The hard part: keeping experts busy
If the router keeps picking the same few experts, the others sit unused and the extra size is wasted. Each paper added fixes for this:
- Extra training penalties when experts are used unevenly (Shazeer, 2017).
- Limits on how many words each expert can take at once (GShard and Switch).
- Simpler routing that sends each word to just one expert (Switch).
Training also has to spread experts across many chips and pass words between them, so this is as much an engineering problem as a design one.
Results that moved
Each paper used different tests, so these figures are milestones rather than a single scoreboard. All come from the paper abstracts.
| Paper | Headline result |
|---|---|
| Shazeer 2017 | Over 1,000 times more model capacity with only small losses in efficiency, up to 137 billion parameters |
| GShard 2020 | A translation model over 600 billion parameters, trained on 2,048 Google chips in 4 days, covering 100 languages |
| Switch 2021 | Up to 7 times faster training than a comparable standard model, and models up to a trillion parameters |
| GLaM 2021 | 1.2 trillion parameters, using a third of the energy needed to train GPT-3 and half the computing per word |
| Mixtral 2024 | 47 billion parameters in total, but only 13 billion used per word |
Mixtral brought the idea to openly available models. It has eight experts per block and sends each word to two of them. Mistral reports that it matches or beats Meta's Llama 2 70B and OpenAI's GPT-3.5 on the benchmarks they tested, and released it under a free Apache 2.0 licence.
What "sparse" means on a bill
A model like Mixtral has two size numbers: 47 billion in total and 13 billion in use for each word. The smaller number roughly sets the processing cost per word. The bigger number sets how much memory you need to run it, because all the experts still have to be loaded.
For businesses using an AI service, that difference is usually hidden in the price. For anyone running a model on their own hardware, it matters a lot. A Mixture-of-Experts model may be cheap per word but still need expensive, high-memory equipment.
Limits
- Training can be unstable, and experts can go unused if routing isn't balanced.
- Running these models efficiently is harder. With only a few requests at a time, the savings can disappear.
- Experts don't neatly specialise in one language or topic. The router's choices are often messier than popular articles suggest.
- More capacity doesn't create knowledge. A model still needs good data, including local data, to be useful in the Caribbean.
Who should care
If you compare AI providers, look at price and speed at your typical prompt length, not the headline parameter count. Our FlashAttention digest covers the other big factor in running costs.
If you plan to host a model yourself, check memory needs for the total size, not just the active size.
For Caribbean AI projects, such as Jamaica's Amber lab, Mixture-of-Experts can stretch a limited computing budget. It won't make a model understand local language on its own. That still needs local data, as our Creole language digest explains.
Bottom line
Mixture-of-Experts lets AI models grow much larger while each word only uses a small part of the model. When a provider quotes a model's size, ask how much of it is used per word and what that means for price and speed.