Implicit Regularization of Dropout
- 23 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (6) , 4206-4217
Abstract
It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we present a theoretical derivation of an implicit regularization of dropout, which is validated by a series of experiments. Additionally, we numerically study two implications of the implicit regularization, which intuitively rationalizes why dropout helps generalization. First, we find that input weights of hidden neurons tend to condense on isolated orientations trained with dropout. Condensation is a feature in the non-linear learning process, which makes the network less complex. Second, we find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training, and the implicit regularization is the key to finding flat solutions. Although our theory mainly focuses on dropout used in the last hidden layer, our experiments apply to general dropout in training neural networks. This work points out a distinct characteristic of dropout compared with stochastic gradient descent and serves as an important basis for fully understanding dropout.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022YFA1008200)
- National Natural Science Foundation of China (92270001, 12371511)
- Shanghai Municipal of Science and Technology Major (2021SHZDZX0102)
- HPC of School of Mathematical Sciences
- Student Innovation Center
- Siyuan-1 cluster
- Center for High Performance Computing at Shanghai Jiao Tong University
This publication has 12 references indexed in Scilit:
- Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural NetworksCSIAM Transactions on Applied Mathematics, 2024
- Towards Understanding the Spectral Bias of Deep Learning2021
- A Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural NetworksChinese Physics Letters, 2021
- Theory of the Frequency Principle for General Deep Neural NetworksCSIAM Transactions on Applied Mathematics, 2021
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural NetworksCommunications in Computational Physics, 2020
- On the Regularization Properties of Structured Dropout2020
- Dropout as an Implicit Gating Mechanism For Continual Learning2020
- Training Behavior of Deep Neural Network in Frequency Domain2019
- Deep Residual Learning for Image Recognition2016
- Geometric numerical integration illustrated by the Störmer–Verlet methodActa Numerica, 2003