Meta Module Network for Compositional Visual Reasoning
Authors:
- 1 January 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 655-664
Abstract
Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical draw-backs: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github 1 .Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- More Than An Answer2017
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning2017
- Scene Graph Generation by Iterative Message Passing2017
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Stacked Attention Networks for Image Question Answering2016
- Visual7W: Grounded Question Answering in Images2016
- Neural Module Networks2016
- Learning to Compose Neural Networks for Question Answering2016
- VQA: Visual Question Answering2015
- Glove: Global Vectors for Word Representation2014