The Advance of Deep Learning and Attention Mechanism
Author:
- 1 September 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 318-321
Abstract
Due to the significant increment of computing power in recent years, research related to machine learning developed and changed dramatically. A new learning model named Transformer, based on an attention mechanism, was introduced by the research team at Google a few years ago. It not only has incredibly high performance but also united two major research fields of machine learning, computer vision, and natural language processing for the first time. However, behind its remarkable success in implementation, there are still potential issues that underlie it. This work is focused on the attention mechanism, for its dominance in the area predominantly affects how machine learning progresses in the future. This paper reviews the development of attention, and its functions, along with many recent models. Besides, this paper will discuss issues related to attention mechanisms and their future directions.Keywords
This publication has 10 references indexed in Scilit:
- Attention Is All You NeedarXiv, 2023
- Masked Autoencoders Are Scalable Vision Learners2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScalearXiv, 2021
- Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) networkPhysica D: Nonlinear Phenomena, 2020
- Recent advances in convolutional neural networksPattern Recognition, 2018
- Deep learning in agriculture: A surveyComputers and Electronics in Agriculture, 2018
- Deep Multimodal Learning: A Survey on Recent Advances and TrendsIEEE Signal Processing Magazine, 2017
- Deep learning for visual understanding: A reviewNeurocomputing, 2016
- Deep Learning: Methods and ApplicationsFoundations and Trends® in Signal Processing, 2014