Meta Module Network for Compositional Visual Reasoning
Authors:
- Wenhu ChenWCWenhu Chen
- Zhe GanZGZhe Gan
- William WangWWWilliam Wang
Preprint
Abstract
Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical drawbacks: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github\footnote{\url{https://github.com/wenhuchen/Meta-Module-Network}}.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Explainable neural computation via stack neural module networksApplied AI Letters, 2021
- MUREL: Multimodal Relational Reasoning for Visual Question Answering2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Improved Fusion of Visual and Language Representations by Dense Symmetric Co-attention for Visual Question Answering2018
- DVQA: Understanding Data Visualizations via Question Answering2018
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning2017
- Scene Graph Generation by Iterative Message Passing2017
- Stacked Attention Networks for Image Question Answering2016
- Visual7W: Grounded Question Answering in Images2016