RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
Preprint
Abstract
We propose a novel and efficient training method for RNNs by iteratively seeking a local minima on the loss surface within a small region, and leverage this directional vector for the update, in an outer-loop. We propose to utilize the Frank-Wolfe (FW) algorithm in this context. Although, FW implicitly involves normalized gradients, which can lead to a slow convergence rate, we develop a novel RNN training method that, surprisingly, even with the additional cost, the overall training cost is empirically observed to be lower than back-propagation. Our method leads to a new Frank-Wolfe method, that is in essence an SGD algorithm with a restart scheme. We prove that under certain conditions our algorithm has a sublinear convergence rate of $O(1/\epsilon)$ for $\epsilon$ error. We then conduct empirical experiments on several benchmark datasets including those that exhibit long-term dependencies, and show significant performance improvement. We also experiment with deep RNN architectures and show efficient training performance. Finally, we demonstrate that our training method is robust to noisy data.
Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- Optimization and applications of echo state networks with leaky- integrator neuronsNeural Networks, 2007
- Dynamical analysis of the brain-state-in-a-box (BSB) neural modelsIEEE Transactions on Neural Networks, 1992
- Stability conditions for nonlinear continuous neural networks with asymmetric connection weightsNeural Networks, 1992
- Stability in contractive nonlinear neural networksIEEE Transactions on Biomedical Engineering, 1990
- A Learning Algorithm for Continually Running Fully Recurrent Neural NetworksNeural Computation, 1989
- On the stability, storage capacity, and design of nonlinear continuous neural networksIEEE Transactions on Systems, Man, and Cybernetics, 1988
- A Trust Region Algorithm for Nonlinearly Constrained OptimizationSIAM Journal on Numerical Analysis, 1987
- Absolute stability of global pattern formation and parallel memory storage by competitive neural networksIEEE Transactions on Systems, Man, and Cybernetics, 1983