Automatic Layer Freezing for Communication Efficiency in Cross-Device Federated Learning
- 29 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 11 (4) , 6072-6083
Abstract
Federated Learning (FL) is a collaborative machine learning paradigm where network-edge clients train a global model under the orchestration of a central server. Unlike traditional distributed learning, each participating client keeps its data locally, ensuring privacy protection by default. However, state-of-the-art FL implementations suffer from massive information exchange between clients and the server. This issue prevents the adoption in constrained environments, typical of the Internet-of-Things domain, where the communication bandwidth and the energy budget are severely limited. To achieve higher efficiency at scale, the future of FL calls for additional optimizations to reach high-quality learning capability with lower communication pressure. To address this challenge, we propose Automatic Layer Freezing (ALF), an embedded mechanism that gradually drops a growing portion of the model out of the training and synchronization phases of the learning loop, reducing the volume of exchanged data with the central server. ALF monitors the evolution of model updates and identifies layers that have reached a stable representation, where further weight updates would have minimal impact on accuracy. By freezing these layers, ALF achieves substantial savings in communication bandwidth and energy consumption. The proposed implementation of the ALF mechanism is compatible with any FL strategy, requiring minimal effort and without interfering with existing optimizations. The extensive experiments conducted using a representative set of FL strategies applied to two image classification tasks show that ALF improves the communication efficiency of the baseline FL implementations, ensuring up to 83.91% of data volume savings with no or marginal losses of accuracy.Keywords
This publication has 28 references indexed in Scilit:
- Communication-Efficient Federated Learning for Wireless Edge Intelligence in IoTIEEE Internet of Things Journal, 2019
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Optimality Assessment of Memory-Bounded ConvNets Deployed on Resource-Constrained RISC CoresIEEE Access, 2019
- Inference on the Edge: Performance Analysis of an Image Classification Task Using Off-The-Shelf CPUs and Open-Source ConvNets2019
- CMFL: Mitigating Communication Overhead for Federated Learning2019
- Adaptive Federated Learning in Resource Constrained Edge Computing SystemsIEEE Journal on Selected Areas in Communications, 2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Layer Skip Learning using LARS variables for 39% Faster Conversion Time and Lower Bandwidth2018
- Weak-MAC: Arithmetic Relaxation for Dynamic Energy-Accuracy Scaling in ConvNets2018
- A close examination of performance and power characteristics of 4G LTE networks2012