Generalization and Risk Bounds for Recurrent Neural Networks
Preprint
Abstract
Recurrent Neural Networks (RNNs) have achieved great success in the prediction of sequential data. However, their theoretical studies are still lagging behind because of their complex interconnected structures. In this paper, we establish a new generalization error bound for vanilla RNNs, and provide a unified framework to calculate the Rademacher complexity that can be applied to a variety of loss functions. When the ramp loss is used, we show that our bound is tighter than the existing bounds based on the same assumptions on the Frobenius and spectral norms of the weight matrices and a few mild conditions. Our numerical results show that our new generalization bound is the tightest among all existing bounds in three public datasets. Our bound improves the second tightest one by an average percentage of 13.80% and 3.01% when the $\tanh$ and ReLU activation functions are used, respectively. Moreover, we derive a sharp estimation error bound for RNN-based estimators obtained through empirical risk minimization (ERM) in multi-class classification problems when the loss function satisfies a Bernstein condition.
Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Recent advances on loss functions in deep learning for computer visionNeurocomputing, 2022
- DeepSleepNet-Lite: A Simplified Automatic Sleep Stage Scoring Model With Uncertainty EstimatesIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021
- Model complexity of deep learning: a surveyKnowledge and Information Systems, 2021
- Robust hierarchical feature selection with a capped -normNeurocomputing, 2021
- A new weighting k -means type clustering framework with an l 2 -norm regularizationKnowledge-Based Systems, 2018
- Local Rademacher complexitiesThe Annals of Statistics, 2005
- Vapnik-Chervonenkis dimension of recurrent neural networksDiscrete Applied Mathematics, 1998
- Sample complexity for learning recurrent perceptron mappingsIEEE Transactions on Information Theory, 1996
- A theory of the learnableCommunications of the ACM, 1984
- On the Uniform Convergence of Relative Frequencies of Events to Their ProbabilitiesTheory of Probability & Its Applications, 1971