How To Loop Moe
Our preprint How to Loop MoE: Flatten the Experts, Untie the Attention is now available on arXiv! We introduce Foil, which flattens expert layers and unties attention to improve looped MoE models at fixed parameters and compute. 🚀
Our preprint How to Loop MoE: Flatten the Experts, Untie the Attention is now available on arXiv! We introduce Foil, which flattens expert layers and unties attention to improve looped MoE models at fixed parameters and compute. 🚀