RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration
Authors:
- 19 October 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5930-5937
Abstract
Open-set semantic mapping is crucial for openworld robots. Current mapping approaches either are limited by the depth range or only map beyond-range entities in constrained settings, where overall they fail to combine within-range and beyond-range observations. Furthermore, these methods make a trade-off between fine-grained semantics and efficiency. We introduce RayFronts, a unified representation that enables both dense and beyond-range efficient semantic mapping. RayFronts encodes task-agnostic openset semantics to both in-range voxels and beyond-range rays encoded at map boundaries, empowering the robot to reduce search volumes significantly and make informed decisions both within & beyond sensory range, while running at 8.84 Hz on an Orin AGX. Benchmarking the within-range semantics shows that RayFronts’s fine-grained image encoding provides 1.34× zero-shot 3D semantic segmentation performance while improving throughput by 16.5×. Traditionally, online mapping performance is entangled with other system components, complicating evaluation. We propose a planner-agnostic evaluation framework that captures the utility for online beyond-range search and exploration, and show RayFronts reduces search volume 2.2× more efficiently than the closest online baselines.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One2024
- Sigmoid Loss for Language Image Pre-Training2023
- Segment Anything2023
- OpenScene: 3D Scene Understanding with Open Vocabularies2023
- Extract Free Dense Labels from CLIP2022
- MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM2019
- Fusion++: Volumetric Object-Level SLAM2018
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017
- SemanticFusion: Dense 3D semantic mapping with convolutional neural networks2017