MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Authors:
- Hedi Ben-YounesHBHedi Ben-Younes
- Remi CadeneRCRemi Cadene
- Matthieu CordMCMatthieu Cord
- Nicolas ThomeNTNicolas Thome
Top Cited Papers
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2631-2639
Abstract
Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how the Tucker decomposition framework generalizes some of the latest VQA architectures, providing state-of-the-art results.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Learning to Compose Neural Networks for Question Answering2016
- Bilinear CNN Models for Fine-Grained Visual Recognition2015
- Multimodal Convolutional Neural Networks for Matching Image and Sentence2015
- VQA: Visual Question Answering2015
- Show and tell: A neural image caption generator2015
- Deep correlation for matching images and text2015
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- Tensor Decompositions and ApplicationsSIAM Review, 2009
- Finding Frequent Items in Data Streams2002
- Some Mathematical Notes on Three-Mode Factor AnalysisPsychometrika, 1966