Neural Network Training Acceleration Based on Hybrid Data and Model Parallelism
Open Access
- 13 January 2026
- journal article
- Published by Ediciones Universidad de Salamanca in ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal
- Vol. 14, e32609
Abstract
Nowadays, deep learning models are quickly increasing in complexity and size, making the training or inference phases unsuitable for most office computers or even high-performance computing servers. Examples of such models include, but are not limited to, large language models and image diffusion models, where the number of parameters to adjust can reach hundreds of billions. Although deep learning models are extremely useful tools, the need for powerful computers to train and use them might compromise the obtained results and also reduce accessibility for many researchers. In this work, we propose a general method to reduce the training time and computing resources (central processing unit and memory usage) by the division of a large-size model into several smaller submodels that are easier to handle but do not affect the final performance. This division depends on the original model and architecture, and hence we propose specific strategies for regression and classification problems. The main result of this work is the development of a public Python library called Skynnet to offer this alternative to deep learning computations.Keywords
This publication has 25 references indexed in Scilit:
- Survey of convolutional neural network accelerators on field-programmable gate array platforms: architectures and optimization techniquesJournal of Real-Time Image Processing, 2024
- Pruning and quantization for deep neural network acceleration: A surveyNeurocomputing, 2021
- Dynamically Distributing Tasks from an Unattended Parallel Compiler with Cloudbook2021
- Training deep neural networks: a static load balancing approachThe Journal of Supercomputing, 2020
- Distributed machine learning load balancing strategy in cloud computing servicesWireless Networks, 2019
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- ImageNet: A large-scale hierarchical image database2009
- The Unreasonable Effectiveness of DataIEEE Intelligent Systems, 2009
- How I came up with the discrete cosine transformDigital Signal Processing, 1991