Align R-CNN: A Pairwise Head Network for Visual Relationship Detection
- 26 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 1266-1276
Abstract
Scene graphs connect individual objects with visual relationships. They serve as a comprehensive scene representation for downstream multimodal tasks. However, by exploring recent progress in Scene Graph Generation (SGG), we find that the performance of recent works is highly limited by the pairwise relationship modeling by naive feature concatenation. Such pairwise features lack sufficient object interaction due to the mis-aligned object parts, resulting in non-discriminative pairwise features for visual relationship prediction. For example, nave concatenated pairwise feature usually make the model fail to discriminate between riding and feeding for object pair person and horse. To this end, we design a meta-architecture learning-to-align for dynamic object feature concatenation. We call our model: Align R-CNN. Specifically, we introduce a novel attention-based multiple region alignment module that can be jointly optimized with SGG. Experiments on the large-scale SGG benchmark Visual Genome show that the proposed Align R-CNN can replace the naive feature concatenation and thus boost all the existing SGG methods.Keywords
This publication has 52 references indexed in Scilit:
- Video Captioning With Attention-Based LSTM and Semantic ConsistencyIEEE Transactions on Multimedia, 2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Visual Relationship Detection with Language Priors2016
- Deep Visual-Semantic Alignments for Generating Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- VQA: Visual Question Answering2015
- Image retrieval using scene graphs2015
- Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval2015
- Combining Language and Vision with a Multimodal Skip-gram Model2015
- NOVELNEURALMODULATORSAnnual Review of Neuroscience, 2003