EMO: New MoE Architecture Enables Domain-Specific Expert Selection for LLMs

Article Summary

EMO presents a significant advancement in Mixture-of-Experts (MoE) architecture, tackling the inherent challenge where standard MoEs fail to specialize experts sufficiently for selective use. Unlike prior methods relying on predefined semantic domains, EMO emerges modularity end-to-end by using document boundaries as a weak supervisory signal during pretraining. This forces the model's router to restrict tokens within a single document to a shared pool of experts, encouraging natural, coherent grouping. The reported model (1B active, 14B total parameters) demonstrates that by activating only 12.5% of its experts, it maintains performance near that of the full model, crucially enabling highly efficient, task-specific deployment without performance degradation. This makes large, sparse MoEs genuinely composable.

Key Points

• EMO addresses the core limitation of standard MoEs by training the architecture with modularity as a first-class objective, allowing experts to naturally group by domain or capability.
• The model's key innovation is restricting token routing within a document to a shared 'expert pool,' which effectively encourages domain-specific specialization and improves selective usability.
• Testing shows that even when utilizing only 12.5% of its total experts, EMO loses only a marginal amount of performance compared to the full model, proving its composability for resource-constrained deployments.