AFSD: Adaptive Feature Space Distillation for Distributed Deep Learning
Open Access
- 8 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 84569-84578
Abstract
We propose a novel and adaptive feature space distillation method (AFSD) to reduce the communication overhead among distributed computers. The proposed method improves the Codistillation process by supporting longer update interval rates. AFSD performs knowledge distillates across the models infrequently and provides flexibility to the models in terms of exploring diverse variations in the training process. We perform knowledge distillation in terms of sharing the feature space instead of output only. Therefore, we also propose a new loss function for the Codistillation technique in AFSD. Using the feature space leads to more efficient knowledge transfer between models with a longer update interval rates. In our method, the models can achieve the same accuracy as Allreduce and Codistillation with fewer epochs.Keywords
Funding Information
- Centre for Integrated Remote Sensing and Forecasting for Arctic Operations (CIRFA) and the Research Council of Norway (237906)
- European Union’s Horizon 2020 Research and Innovation Programme Extreme Earth Project (825258)
- Fram Center under the Automised Large-Scale Sea Ice Mapping (ALSIM) through the “Polhavet” Flagship Project
This publication has 17 references indexed in Scilit:
- Cost-Aware Partitioning for Efficient Large Graph Processing in Geo-Distributed DatacentersIEEE Transactions on Parallel and Distributed Systems, 2019
- SparCML2019
- Demystifying Parallel and Distributed Deep LearningACM Computing Surveys, 2019
- Label Propagation for Deep Semi-Supervised Learning2019
- Deep Mutual Learning2018
- Efficient Knowledge Distillation from an Ensemble of Teachers2017
- S-Caffe2017
- Deep Residual Learning for Image Recognition2016
- GeePS2016
- A parallel generalized relaxation method for high-performance image segmentation on GPUsJournal of Computational and Applied Mathematics, 2016