Learning 3D Semantic Scene Graphs with Instance Embeddings
Open Access
- 22 January 2022
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 130 (3) , 630-651
Abstract
A 3D scene is more than the geometry and classes of the objects it comprises. An essential aspect beyond object-level perception is the scene context, described as a dense semantic network of interconnected nodes. Scene graphs have become a common representation to encode the semantic richness of images, where nodes in the graph are object entities connected by edges, so-called relationships. Such graphs have been shown to be useful in achieving state-of-the-art performance in image captioning, visual question answering and image generation or editing. While scene graph prediction methods so far focused on images, we propose instead a novel neural network architecture for 3D data, where the aim is to learn to regress semantic graphs from a given 3D scene. With this work, we go beyond object-level perception, by exploring relations between object entities. Our method learns instance embeddings alongside a scene segmentation and is able to predict semantics for object nodes and edges. We leverage 3DSSG, a large scale dataset based on 3RScan that features scene graphs of changing 3D scenes. Finally, we show the effectiveness of graphs as an intermediate representation on a retrieval task.Keywords
Funding Information
- Technische Universität München
This publication has 79 references indexed in Scilit:
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- 24/7 place recognition by view synthesis2015
- Image retrieval using scene graphs2015
- SUN RGB-D: A RGB-D scene understanding benchmark suite2015
- BiggerPictureACM Transactions on Graphics, 2014
- Creating consistent scene graphs using a probabilistic grammarACM Transactions on Graphics, 2014
- Real-Time RGB-D Camera Relocalization via Randomized Ferns for Keyframe EncodingIEEE Transactions on Visualization and Computer Graphics, 2014
- Indoor Segmentation and Support Inference from RGBD Images2012
- Characterizing structural relationships in scenes using graph kernels2011
- A survey of content-based image retrieval with high-level semanticsPattern Recognition, 2007