Report for: Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Upcycling serves as an efficient paradigm for initializing MoE models by repurposing pre-trained dense models. By leveraging the rich representations already encoded within a dense model, this approach substantially reduces computational overhead while achieving performance comparable to that of MoE models trained from scratch. Our method, unlike naive upcycling, uses a sub-matrix splitting technique to initialize fine-grained experts, combined with Drop-Upcycling to catalyze specialized expert convergence and minimize redundancy bottlenecks. This facilitates expert diversification necessary for nuanced multilingual reasoning.

Marco-MoE is pre-trained on a large amount of high-quality multilingual data sourced from the web and synthesized using various strategies. We aggregate web-crawled data, prioritizing high-quality variants and employing a rephrasing strategy for noisier content. Synthesized data includes multilingual QA data and STEM data, generated through translation and specialized curation to enhance model performance across diverse benchmarks, especially for low-resource languages and culturally salient knowledge.

Our Marco-MoE base models demonstrate a superior performance-to-compute ratio and set the state-of-the-art for simultaneous proficiency in both English and multilingual capabilities. Marco-Nano-Instruct and Marco-Mini-Instruct consistently match or outperform models with 3–14× more activated parameters, despite activating only a fraction of the total parameters. This efficiency is maintained across various benchmarks, highlighting a best-in-class performance-to-compute ratio.