SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14531-14542
Abstract
Existing 3D scene understanding methods are heavily focused on 3D semantic and instance segmentation. However, identifying objects and their parts only constitutes an intermediate step towards a more fine-grained goal, which is effectively interacting with the functional interactive elements (e.g., handles, knobs, buttons) in the scene to accomplish diverse tasks. To this end, we introduce SceneFun3D, a large-scale dataset with more than 14. Bk highly accurate interaction annotations for 710 high-resolution real-world 3D indoor scenes. We accompany the annotations with motion parameter information, describing how to interact with these elements, and a diverse set of natural language descriptions of tasks that involve manipulating them in the scene context. To showcase the value of our dataset, we introduce three novel tasks, namely functionality segmentation, task-driven affordance grounding and 3D motion estimation, and adapt existing state-of-the-art methods to tackle them. Our experiments show that solving these tasks in real 3D scenes remains challenging despite recent progress in closed-set and open-set 3D scene understanding methods.Keywords
This publication has 57 references indexed in Scilit:
- 4D-StOP: Panoptic Segmentation of 4D LiDAR Using Spatio-Temporal Object Proposal Generation and Aggregation2023
- Unsupervised Pose-aware Part Decomposition for Man-Made Articulated Objects2022
- Language-Grounded Indoor 3D Semantic Segmentation in the Wild2022
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- OPD: Single-View 3D Openable Part Detection2022
- SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans2020
- ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language2020
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- 3D Bird’s-Eye-View Instance Segmentation2019
- SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters2018