Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation
Preprint
Abstract
Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. Language-specific modeling methods show promise in reducing interference. However, they often rely on heuristics to distribute capacity and struggle to foster cross-lingual transfer via isolated modules. In this paper, we explore intrinsic task modularity within multilingual networks and leverage these observations to circumvent interference under multilingual translation. We show that neurons in the feed-forward layers tend to be activated in a language-specific manner. Meanwhile, these specialized neurons exhibit structural overlaps that reflect language proximity, which progress across layers. Based on these findings, we propose Neuron Specialization, an approach that identifies specialized neurons to modularize feed-forward layers and then continuously updates them through sparse networks. Extensive experiments show that our approach achieves consistent performance gains over strong baselines with additional analyses demonstrating reduced interference and increased knowledge transfer.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Neurons in Large Language Models: Dead, N-gram, Positional2024
- Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks2024
- Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency ParsingComputational Linguistics, 2023
- Emergent Modularity in Pre-trained Transformers2023
- Parameter-Efficient Fine-Tuning without Introducing New Latency2023
- Brain-like functional specialization emerges spontaneously in deep neural networksScience Advances, 2022
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling2019