Convolution-Enhanced Evolving Attention Networks
- 13 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 45 (7) , 8176-8192
Abstract
Attention-based neural networks, such as Transformers, have become ubiquitous in numerous applications, including computer vision, natural language processing, and time-series analysis. In all kinds of attention networks, the attention maps are crucial as they encode semantic dependencies between input tokens. However, most existing attention networks perform modeling or reasoning based on representations , wherein the attention maps of different layers are learned separately without explicit interactions. In this paper, we propose a novel and generic evolving attention mechanism, which directly models the evolution of inter-token relationships through a chain of residual convolutional modules. The major motivations are twofold. On the one hand, the attention maps in different layers share transferable knowledge, thus adding a residual connection can facilitate the information flow of inter-token relationships across layers. On the other hand, there is naturally an evolutionary trend among attention maps at different abstraction levels, so it is beneficial to exploit a dedicated convolution-based module to capture this process. Equipped with the proposed mechanism, the convolution-enhanced evolving attention networks achieve superior performance in various applications, including time-series representation, natural language understanding, machine translation, and image classification. Especially on time-series representation tasks, Evolving Attention-enhanced Dilated Convolutional (EA-DC-) Transformer outperforms state-of-the-art models significantly, achieving an average of 17% improvement compared to the best SOTA. To the best of our knowledge, this is the first work that explicitly models the layer-wise evolution of attention maps. Our implementation is available at https://github.com/pkuyym/EvolvingAttention .Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2020YFB2103402)
- Beijing Academy of Artificial Intelligence
This publication has 47 references indexed in Scilit:
- A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction2017
- Densely Connected Convolutional Networks2017
- Time series classification from scratch with deep neural networks: A strong baseline2017
- "Why Should I Trust You?"2016
- Deep Residual Learning for Image Recognition2016
- Neural Machine Translation of Rare Words with Subword Units2016
- Effective Approaches to Attention-based Neural Machine Translation2015
- Findings of the 2014 Workshop on Statistical Machine Translation2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Long Short-Term MemoryNeural Computation, 1997