Speeding up the Training of Neural Networks with the One-Step Procedure
Open Access
- 21 May 2024
- journal article
- research article
- Published by Springer Nature in Neural Processing Letters
- Vol. 56 (3) , 1-13
Abstract
In the last decade, research and corporate have shown a dramatically growing interest in the field of machine learning, mostly due to the performances of deep neural networks. These increasingly complex architectures solved a wide range of problems. However, training these sophisticated architectures require many computation on advanced hardware. With this paper, we introduce a new approach based on the One-Step procedure that may fasten their training. In this procedure, an initial guess estimator is computed on a subsample that is then improved with only one step of the Newton gradient descent on the whole dataset. To show the efficiency of this framework, we consider regression and classification tasks using simulated and real datasets. We consider classic architectures, namely multi-layer perceptrons and show, on our examples, that the One-Step procedure is often halving the computation time to train the neural networks while preserving the performances.Keywords
Funding Information
- Agence Nationale de la Recherche (ANR-21-CE40-0021, ANR-21-CE23-0010)
This publication has 22 references indexed in Scilit:
- Scaling Vision Transformers2022
- Neural network approximation: Three hidden layers are enoughNeural Networks, 2021
- Green Algorithms: Quantifying the Carbon Footprint of ComputationAdvanced Science, 2021
- DeepXDE: A Deep Learning Library for Solving Differential EquationsSIAM Review, 2021
- One-step estimation for the fractional Gaussian noise at high-frequencyESAIM: Probability and Statistics, 2020
- Fast Generalized Linear Models by Database Sampling and One-Step PolishingJournal of Computational and Graphical Statistics, 2019
- Efficient estimation of stable Lévy process with symmetric jumpsStatistical Inference for Stochastic Processes, 2018
- On multi-step MLE-process for Markov sequencesMetrika, 2015
- Calibrating Probability with Undersampling for Unbalanced Classification2015
- Hybrid multi-step estimators for stochastic differential equations based on sampled dataStatistical Inference for Stochastic Processes, 2014