Motif-3

We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13. 2 billion parameters activated for each token. Each sparse MoE layer contains 384 routed experts, while only eight are selected for each token.

Running this yourself: likely needs a high-memory cloud gpu.