Multi-Modality Global Fusion Attention Network for Visual Question Answering
Open Access
- 9 November 2020
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 9 (11) , 1882
Abstract
Visual question answering (VQA) requires a high-level understanding of both questions and images, along with visual reasoning to predict the correct answer. Therefore, it is important to design an effective attention model to associate key regions in an image with key words in a question. Up to now, most attention-based approaches only model the relationships between individual regions in an image and words in a question. It is not enough to predict the correct answer for VQA, as human beings always think in terms of global information, not only local information. In this paper, we propose a novel multi-modality global fusion attention network (MGFAN) consisting of stacked global fusion attention (GFA) blocks, which can capture information from global perspectives. Our proposed method computes co-attention and self-attention at the same time, rather than computing them individually. We validate our proposed method on the two most commonly used benchmarks, the VQA-v2 datasets. Experimental results show that the proposed method outperforms the previous state-of-the-art. Our best single model achieves 70.67% accuracy on the test-dev set of VQA-v2.Keywords
Funding Information
- National Key Research and Development Program of China (2019YFC0118200)
This publication has 26 references indexed in Scilit:
- Leveraging unpaired out-of-domain data for image captioningPattern Recognition Letters, 2020
- Multimodal feature fusion by relational reasoning and attention for visual question answeringInformation Fusion, 2020
- Biometric surveillance using visual question answeringPattern Recognition Letters, 2019
- Deep Modular Co-Attention Networks for Visual Question Answering2019
- Image caption generation with high-level image featuresPattern Recognition Letters, 2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge2018
- Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question AnsweringIEEE Transactions on Neural Networks and Learning Systems, 2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Stacked Attention Networks for Image Question Answering2016