3D Feature Distillation with Object-Centric Priors
Preprint
- 17 October 2025
- preprint
- Published by Springer Nature in Research Square
Abstract
Grounding natural language to the physical world is a ubiquitous topic with a widerange of applications in computer vision and robotics. Recently, 2D vision-languagemodels such as CLIP have been widely popularized, due to their impressivecapabilities for open-vocabulary grounding in 2D images. Subsequent works aimto elevate 2D CLIP features to 3D via feature distillation, but either learn neuralfields that are scene-specific and hence lack generalization, or focus on indoor roomscan data that require access to multiple camera views, which is not practical inrobot manipulation scenarios. Additionally, related methods typically fuse featuresat pixel-level and assume that all camera views are equally informative. In thiswork, we show that this approach leads to sub-optimal 3D features, both in termsof grounding accuracy, as well as segmentation crispness. To alleviate this, wepropose a multi-view feature fusion strategy that employs object-centric priors toeliminate uninformative views based on semantic information, and fuse featuresat object-level via instance segmentation masks. To distill our object-centric3D features, we generate a large-scale synthetic multi-view dataset of clutteredtabletop scenes, spawning 15k scenes from over 3300 unique object instances,which we make publicly available. We show that our method reconstructs 3DCLIP features with improved grounding capacity and spatial consistency, whiledoing so from single-view RGB-D, thus departing from the assumption of multiplecamera views at test time. Finally, we show that our approach can generalize tonovel tabletop domains and be re-purposed for 3D instance segmentation withoutfine-tuning, and demonstrate its utility for language-guided robotic grasping in clutter. Released assets and supplementary material is made available at thewebsite https://gtziafas.github.io/DROP_project/.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- OpenIns3D: Snap and Lookup for 3D Open-Vocabulary Instance Segmentation2024
- RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding2024
- MaskClustering: View Consensus Based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation2024
- LangSplat: 3D Language Gaussian Splatting2024
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance2024
- SAI3D: Segment any Instance in 3D Scenes2024
- Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships2024
- Detecting Twenty-Thousand Classes Using Image-Level Supervision2022
- Language-Grounded Indoor 3D Semantic Segmentation in the Wild2022
- ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language2020