Effects of depth, width, and initialization: A convergence analysis of layer-wise training for deep linear neural networks
Author:
- 31 December 2021
- journal article
- research article
- Published by World Scientific Pub Co Pte Ltd in Analysis and Applications
- Vol. 20 (01) , 73-119
Abstract
Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging task. Many alternatives have been proposed in place of end-to-end back-propagation. Layer-wise training is one of them, which trains a single layer at a time, rather than trains the whole layers simultaneously. In this paper, we study a layer-wise training using a block coordinate gradient descent (BCGD) for deep linear networks. We establish a general convergence analysis of BCGD and found the optimal learning rate, which results in the fastest decrease in the loss. We identify the effects of depth, width, and initialization. When the orthogonal-like initialization is employed, we show that the width of intermediate layers plays no role in gradient-based training beyond a certain threshold. Besides, we found that the use of deep networks could drastically accelerate convergence when it is compared to those of a depth 1 network, even when the computational cost is considered. Numerical examples are provided to justify our theoretical findings and demonstrate the performance of layer-wise training by BCGD.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- On the calibration of sensor arrays for pattern recognition using the minimal number of experimentsChemometrics and Intelligent Laboratory Systems, 2014
- Multi-column deep neural networks for image classification2012
- Chemical gas sensor drift compensation using classifier ensemblesSensors and Actuators B: Chemical, 2012
- Randomized Methods for Linear Constraints: Convergence Rates and ConditioningMathematics of Operations Research, 2010
- Randomized Kaczmarz solver for noisy linear systemsBIT Numerical Mathematics, 2010
- Block-Coordinate Gradient Descent Method for Linearly Constrained Nonsmooth Separable OptimizationJournal of Optimization Theory and Applications, 2008
- A Randomized Kaczmarz Algorithm with Exponential ConvergenceJournal of Fourier Analysis and Applications, 2008
- A coordinate gradient descent method for nonsmooth separable minimizationMathematical Programming, 2007
- Reducing the Dimensionality of Data with Neural NetworksScience, 2006
- Training MLPs layer by layer using an objective function for internal representationsNeural Networks, 1996