AI Glossary
Mixture of Experts (MoE)
What is Mixture of Experts (MoE)?
A neural network architecture in which different parts of the model ('experts') specialise in different types of input, with a routing mechanism selecting which experts to activate for each token. MoE architectures allow very large effective model capacity while keeping per-token compute cost manageable. GPT-4 is widely believed to use a MoE architecture.
Example in practice
An enterprise that notices GPT-4 performs significantly better on legal text than a same-size dense model may be benefiting from MoE specialisation — where different expert sub-networks have developed stronger domain representations.
Learn more
See Mixture of Experts (MoE) applied in a professional context through this free course.
AI Fundamentals for Professionals →