MASS: MoErging through Adaptive Subspace Selection

@misc{crisostomi2025massmoergingadaptivesubspace,
      title={MASS: MoErging through Adaptive Subspace Selection}, 
      author={Donato Crisostomi and Alessandro Zirilli and Antonio Andrea Gargiulo and Maria Sofia Bucarelli and Simone Scardapane and Fabrizio Silvestri and Iacopo Masi and Emanuele Rodolà},
      year={2025},
      eprint={2504.05342},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2504.05342}, 
}

MASS is our novel, training-free model MoErging method, that recovers up to ~98% of the accuracy of finetuned models with only a two times increase in storage and computational cost.

We tested its versatility across different domains (Vision and NLP), architectures (ViT-{B,L}-{32,16,14}, Flan-t5), and number of tasks (8, 14, and 20 datasets), proving its incredible scalability at fixed overhead cost.

At 🤗 this page you can find all the checkpoints you need, while in the card below there is our codebase that contains detailed instructions to reproduce our experiments.