RNEP: Random Node Entropy Pairing for Efficient Decentralized Training with Non-IID Local Data
Open Access
- 25 October 2024
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 13 (21) , 4193
Abstract
The proliferation of edge devices and advancements in Internet of Things technology have created a vast array of distributed data sources, necessitating machine learning models that can operate closer to the point of data generation. Traditional centralized machine learning approaches often struggle with real-time big data applications, such as climate prediction and traffic simulation, owing to high communication costs. In this study, we introduce random node entropy pairing, a novel distributed learning method for artificial neural networks tailored to distributed computing environments. This method reduces communication overhead and addresses data imbalance by enabling nodes to exchange only the weights of their local models with randomly selected peers during each communication round, rather than sharing the entire dataset. Our findings indicate that this approach significantly reduces communication costs while maintaining accuracy, even when learning from non-IID local data. Furthermore, we explored additional learning strategies that enhance system accuracy by leveraging the characteristics of this method. The results demonstrate that random node entropy pairing is an effective and efficient solution for distributed learning in environments where communication costs and data distribution present significant challenges.Keywords
Funding Information
- Ministry of Trade, Industry and Energy (RS-2024-00403397, NRF-2021R1F1A1060183)
- Kwangwoon University (RS-2024-00403397, NRF-2021R1F1A1060183)
- Korea Government (MSIT) (RS-2024-00403397, NRF-2021R1F1A1060183)
This publication has 41 references indexed in Scilit:
- Experimenting With Normalization Layers in Federated Learning on Non-IID ScenariosIEEE Access, 2024
- Entropy to Mitigate Non-IID Data Problem on Federated Learning for the Edge Intelligence EnvironmentIEEE Access, 2023
- SaPus: Self-Adaptive Parameter Update Strategy for DNN Training on Multi-GPU ClustersIEEE Transactions on Parallel and Distributed Systems, 2021
- Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private DataIEEE Transactions on Mobile Computing, 2021
- Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset TransferIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Self-Balancing Federated Learning With Global Imbalanced Data in Mobile SystemsIEEE Transactions on Parallel and Distributed Systems, 2020
- Towards Ubiquitous Intelligent Computing: Heterogeneous Distributed Deep Neural NetworksIEEE Transactions on Big Data, 2018
- A Bi-layered Parallel Training Architecture for Large-Scale Convolutional Neural NetworksIEEE Transactions on Parallel and Distributed Systems, 2018
- Distributed learning for Random Vector Functional-Link networksInformation Sciences, 2015
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948