Mind the Remainder: Taylor’s Theorem View on Recurrent Neural Networks
- 14 January 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 33 (4) , 1507-1519
Abstract
Recurrent neural networks (RNNs) have gained tremendous popularity in almost every sequence modeling task. Despite the effort, these kinds of discrete unstructured data, such as texts, audio, and videos, are still difficult to be embedded in the feature space. Studies in improving the neural networks have accelerated since the introduction of more complex or deeper architectures. The improvements of previous methods are highly dependent on the model at the expense of huge computational sources. However, few of them pay attention to the algorithm. In this article, we bridge the Taylor series with the construction of RNN. Training RNN can be considered as a parameter estimate for the Taylor series. However, we found that there is a discrete term called the remainder in the finite Taylor series that cannot be optimized using gradient descent, which is part of the reason for the truncation error and the model falling into the local optimal solution. To address this, we propose a training algorithm that estimates the range of remainder and introduces the remainder obtained by sampling in this continuous space into the RNN to assist in optimizing the parameters. Notably, the performance of RNN can be improved without changing the RNN architecture in the testing phase. We demonstrate that our approach is able to achieve state-of-the-art performance in action recognition and cross-modal retrieval tasks.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0102200)
- Sichuan Science and Technology Program, China (2018GZDZX0032, 2020YFS0057)
- Fundamental Research Funds for the Central Universities (ZYGX2019Z015)
- National Natural Science Foundation of China (61632007)
- Dongguan Songshan Lake Introduction Program of Leading Innovative and Entrepreneurial Talents
This publication has 32 references indexed in Scilit:
- Actions ~ Transformations2016
- Vector Taylor series based model adaptation using noisy speech trained hidden Markov modelsPattern Recognition Letters, 2016
- Beyond short snippets: Deep networks for video classification2015
- Action recognition with trajectory-pooled deep-convolutional descriptors2015
- Deep visual-semantic alignments for generating image descriptions2015
- Visual Semantic Search: Retrieving Videos via Complex Textual Queries2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Long Short-Term MemoryNeural Computation, 1997
- Approximation capabilities of multilayer feedforward networksNeural Networks, 1991
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989