Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
Authors:
- Jinmian YeJYJinmian Ye
- Linnan WangLWLinnan Wang
- Guangxi LiGLGuangxi Li
- Di ChenDCDi Chen
- Shandian ZheSZShandian Zhe
- Zenglin XuZXZenglin Xu
Preprint
Abstract
Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in Videos and Image Captioning. To overcome this problem, we propose a compact and flexible structure, namely Block-Term tensor decomposition, which greatly reduces the parameters of RNNs and improves their training efficiency. Compared with alternative low-rank approximations, such as tensor-train RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not only more concise (when using the same rank), but also able to attain a better approximation to the original RNNs with much fewer parameters. On three challenging tasks, including Action Recognition in Videos, Image Captioning and Image Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms of both prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes 17,388 times fewer parameters than the standard LSTM to achieve an accuracy improvement over 15.6\% in the Action Recognition task on the UCF11 dataset.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Learning Efficient Tensor Representations with Ring-structured Networks2019
- Accelerating deep neural network training with inconsistent stochastic gradient descentNeural Networks, 2017
- Compressing recurrent neural network with tensor train2017
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Bayesian Nonparametric Models for Multiway Data AnalysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Tensor-Train DecompositionSIAM Journal on Scientific Computing, 2011
- Decompositions of a Higher-Order Tensor in Block Terms—Part II: Definitions and UniquenessSIAM Journal on Matrix Analysis and Applications, 2008
- Learning long-term dependencies with gradient descent is difficultIEEE Transactions on Neural Networks, 1994
- Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” DecompositionPsychometrika, 1970
- Some Mathematical Notes on Three-Mode Factor AnalysisPsychometrika, 1966