Improving Model Fusion by Training-Time Neuron Alignment With Fixed Neuron Anchors
- 22 October 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2186-2199
Abstract
Model fusion aims to integrate several deep neural network (DNN) models’ knowledge into one by fusing parameters, and it has promising applications, such as improving the generalization of foundation models and parameter averaging in federated learning. However, models under different settings (data, hyperparameter, etc.) have diverse neuron permutations; in other words, from the perspective of loss landscape, they reside in different loss basins, thus hindering model fusion performances. To alleviate this issue, previous studies highlighted the role of permutation invariance and have developed methods to find correct network permutations for neuron alignment after training. Orthogonal to previous attempts, this paper studies training-time neuron alignment, improving model fusion without the need for post-matching. Training-time alignment is cheaper than post-alignment and is applicable in various model fusion scenarios. Starting from fundamental hypotheses and theorems, a simple yet lossless algorithm called TNA-PFN is introduced. TNA-PFN utilizes partially fixed neuron weights as anchors to reduce the potential of training-time permutations, and it is empirically validated in reducing the barriers of linear mode connectivity and multi-model fusion. It is also validated that TNA-PFN can improve the fusion of pretrained models under the setting of model soup (vision transformers) and ColD fusion (pretrained language models). Based on TNA-PFN, two federated learning methods, FedPFN and FedPNU, are proposed, showing the prospects of training-time neuron alignment. FedPFN and FedPNU reach state-of-the-art performances in federated learning under heterogeneous settings and can be compatible with the server-side algorithm.Keywords
All Related Versions
Funding Information
- National Science and Technology Major Project (2022ZD0115101)
- Research Center for Industries of the Future
- Westlake University
- Westlake Education Foundation
- National Key Research and Development Project of China (2021ZD0110505)
- Zhejiang Provincial Key Research and Development Project (2023C01043)
- Academy Of Social Governance Zhejiang University
- National Natural Science Foundation of China (62441617)
- Zhejiang Provincial Natural Science Foundation of China (LD25F020001)
- Fundamental Research Funds for the Central Universities (226-2025-00057)
This publication has 16 references indexed in Scilit:
- No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier2023
- Re-basin via implicit Sinkhorn differentiation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Federated Learning with Position-Aware Neurons2022
- Deep Residual Learning for Image Recognition2016
- Supervised Sequence Labelling with Recurrent Neural Networks2012
- ImageNet: A large-scale hierarchical image database2009
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Sparse spatial autoregressionsStatistics & Probability Letters, 1997
- ON THE ALGEBRAIC STRUCTURE OF FEEDFORWARD NETWORK WEIGHT SPACES1990