Relation Transformer Network
Preprint
Abstract
The extraction of a scene graph with objects as nodes and mutual relationships as edges is the basis for a deep understanding of image content. Despite recent advances, such as message passing and joint classification, the detection of visual relationships remains a challenging task due to sub-optimal exploration of the mutual interaction among the visual objects. In this work, we propose a novel transformer formulation for scene graph generation and relation prediction. We leverage the encoder-decoder architecture of the transformer for rich feature embedding of nodes and edges. Specifically, we model the node-to-node interaction with the self-attention of the transformer encoder and the edge-to-node interaction with the cross-attention of the transformer decoder. Further, we introduce a novel positional embedding suitable to handle edges in the decoder. Finally, our relation prediction module classifies the directed relation from the learned node and edge embedding. We name this architecture as Relation Transformer Network (RTN). On the Visual Genome and GQA dataset, we have achieved an overall mean of 4.85% and 3.1% point improvement in comparison with state-of-the-art methods. Our experiments show that Relation Transformer can efficiently model context across various datasets with small, medium, and large-scale relation classification.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering2021
- Improving Visual Relation Detection using Depth Maps2021
- Learning Visual Commonsense for Robust Scene Graph Generation2020
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-TrainingProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned2019
- Scene graph captioner: Image captioning based on structural visual representationJournal of Visual Communication and Image Representation, 2019
- Representation Learning for Scene Graph Completion via Jointly Structural and Visual Embedding2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017