Deep Residual Weight-Sharing Attention Network With Low-Rank Attention for Visual Question Answering
- 6 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 4282-4295
Abstract
The attention-based networks have become prevailing recently in visual question answering (VQA) due to their high performances. However, the extensive memory consumption of attention-based models poses excessive-high demand for the implementation equipment, raising concerns about their future application scenarios. Therefore, designing an efficient and lightweight VQA model is central to expanding possible application areas. Our work presents a novel lightweight attention-based VQA model, namely residual weight-sharing attention network (RWSAN), consisting of residual weight-sharing attention (RWSA) layers cascaded in depth. Each RWSA layer models the textual representation with self residual weight-sharing attention (SRWSA) and captures question features and question-image interactions with self-guided residual weight-sharing attention (SGRWSA). Inside each RWSA layer, the proposed low-rank attention units perform residual learning with learned connection patterns and shared parameters, and every stacked RWSA layer also uses the same parameters. Extensive ablation experiments with quantitative and qualitative analysis are conducted to illustrate the effectiveness and generality of RWSA. Experiments on VQA-v2, GQA, and CLEVR datasets show that the RWSAN achieves competitive performance with much fewer parameters over the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (U21B2004)
- Zhejiang Provincial key RD Program (2021C01119)
This publication has 51 references indexed in Scilit:
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Stacked Attention Networks for Image Question Answering2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- Glove: Global Vectors for Word Representation2014
- An algorithmic theory of learning: Robust concepts and random projectionMachine Learning, 2006
- Long Short-Term MemoryNeural Computation, 1997
- Finding structure in timeCognitive Science, 1990
- Extensions of Lipschitz mappings into a Hilbert space1984