Mixture of Experts
Language Models & ArchitectureA model architecture that contains many specialized sub-networks, or experts, but activates only a few for any given input, so the model can be very large yet cheap to run.
Mixture-of-experts models concentrate their memory demand while keeping per-query compute modest, a balance the large HBM3e capacity of Eleveight AI's B300 GPUs is well suited to hold.
Overview
A mixture of experts is a way to grow a model's capacity without growing the cost of every inference in step. Instead of one monolithic network where all parameters fire for every input, an MoE model holds many expert sub-networks and, for each input, routes the work to just a small handful of them. The model can therefore hold a very large number of total parameters while only a fraction do work on any single request.
How it works
A lightweight component called a router examines each input and selects which experts should handle it, sending the work only to those few. The chosen experts compute their outputs, which are combined into the final result. Because only a small subset is active at a time, the compute per token stays low even as the total parameter count, and with it the model's overall knowledge, climbs.
Why it matters
MoE decouples a model's size from its running cost, which is why several of the largest and most capable recent models use the approach. The trade-off lands on memory: all the experts must be held in GPU memory ready to be called, even though only a few are used per token. This places a premium on memory capacity and bandwidth, exactly where high-end accelerators are strongest.
Use cases
- Scaling model capacity without proportional inference cost
- Large language models with very high parameter counts
- Efficient serving of frontier-scale models
- Specialized expert routing for diverse inputs