Focus Your Attention: A Focal Attention for Multimodal Learning
- 23 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 103-115
Abstract
The key point in multimodal learning is to learn semantic alignment that finds the correspondence between sub-elements of instances from different modality data. Attention mechanism has shown its power in semantic alignment learning as it enables to densely associate sub-elements across different modalities. However, for each sub-element, existing attention aligns it with all the sub-elements from another modality, while most of them have no correspondence with it, i.e. irrelevant sub-elements. The irrelevant sub-elements will distract the semantic alignment if they are also attended. In this paper, we propose a novel focal attention mechanism to learn more accurate semantic alignment. The focal attention sparsely attends to a subset of sub-elements, which are identified as relevant ones according to their posterior probabilities given each sub-element from another modality. Based on the observation that relevant sub-elements mostly describe the same semantic, the posterior probability can precisely distinguish relevant and irrelevant ones by taking interactions within the same modality into consideration, such that relevant sub-elements get higher and closer posterior probabilities, while irrelevant ones get lower probabilities. Such a design learns better semantic alignment by preventing the interference of irrelevant sub-elements, and it facilitates subsequent multimodal tasks that demand semantic alignment. To validate the effectiveness of the focal attention, we conduct extensive experiments on image-text matching and text-to-image generation, and we propose a bidirectional and stacked version of focal attention for them, respectively. Experimental results on benchmarks show that the focal attention can significantly and consistently outperform state-of-the-arts.Keywords
Funding Information
- National Key Research and Development Program of China (2020YFB1406603)
- National Natural Science Foundation of China (U19A2057)
- National Science Fund for Distinguished Young Scholars (61525206)
- Fundamental Research Funds for the Central Universities (WK3480000008)
This publication has 43 references indexed in Scilit:
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Learning Deep Representations of Fine-Grained Visual Descriptions2016
- Rethinking the Inception Architecture for Computer Vision2016
- Stacked Attention Networks for Image Question Answering2016
- Cross-Modal Retrieval via Deep and Bidirectional Representation LearningIEEE Transactions on Multimedia, 2016
- Cross-Modal Correlation Learning by Adaptive Hierarchical Semantic AggregationIEEE Transactions on Multimedia, 2016
- Discriminative Dictionary Learning With Common Label Alignment for Cross-Modal RetrievalIEEE Transactions on Multimedia, 2015
- Multimodal Convolutional Neural Networks for Matching Image and Sentence2015
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015