Mixtral of Experts: The 47B Sparse MoE That Runs Like a 13B Model
Mixtral 8x7B routes each token to 2 of 8 experts per layer, so it holds 47B parameters but uses only ~13B per token — and matches or beats Llama 2 70B and GPT-3.5 under Apache 2.0.
Institution
Mistral AI
European lab shipping efficient open-weight language models such as Mistral 7B and the Mixtral mixture-of-experts.
Mistral 7B: The 7B Open Model That Beat Llama 2 13B
Mistral 7B is a 7-billion-parameter open model that outperforms Llama 2 13B on every benchmark tested, uses grouped-query and sliding-window attention for cheap inference, and ships under Apache 2.0.