Configurable Graph Reasoning for Visual Relationship Detection
- 29 October 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 33 (1) , 117-129
Abstract
Visual commonsense knowledge has received growing attention in the reasoning of long-tailed visual relationships biased in terms of object and relation labels. Most current methods typically collect and utilize external knowledge for visual relationships by following the fixed reasoning path of {subject, object → predicate} to facilitate the recognition of infrequent relationships. However, the knowledge incorporation for such fixed multidependent path suffers from the data set biased and exponentially grown combinations of object and relation labels and ignores the semantic gap between commonsense knowledge and real scenes. To alleviate this, we propose configurable graph reasoning (CGR) to decompose the reasoning path of visual relationships and the incorporation of external knowledge, achieving configurable knowledge selection and personalized graph reasoning for each relation type in each image. Given a commonsense knowledge graph, CGR learns to match and retrieve knowledge for different subpaths and selectively compose the knowledge routed path. CGR adaptively configures the reasoning path based on the knowledge graph, bridges the semantic gap between the commonsense knowledge, and the real-world scenes and achieves better knowledge generalization. Extensive experiments show that CGR consistently outperforms previous state-of-the-art methods on several popular benchmarks and works well with different knowledge graphs. Detailed analyses demonstrated that CGR learned explainable and compelling configurations of reasoning paths.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0100300)
- National Natural Science Foundation of China (61836012, U19A2073, 61976233)
- Guangdong Province Basic and Applied Basic Research (2019B1515120039)
- Nature Science Foundation of Shenzhen (2019191361)
- Zhijiang Laboratory’s Open Fund (2020AA3AB14)
This publication has 30 references indexed in Scilit:
- From Red Wine to Red Tomato: Composition with Context2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- Deep Variation-Structured Reinforcement Learning for Visual Relationship and Attribute Detection2017
- Visual Relationship Detection with Language Priors2016
- Learning Deep Representations of Fine-Grained Visual Descriptions2016
- The Role of Context for Object Detection and Semantic Segmentation in the Wild2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Glove: Global Vectors for Word Representation2014
- Relative attributes2011
- Long Short-Term MemoryNeural Computation, 1997