Learning Semantic Traversability With Egocentric Video and Automated Annotation Strategy
- 3 October 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (11) , 10423-10430
Abstract
For reliable autonomous robot navigation in urban settings, the robot must have the ability to identify semantically traversable terrains in the image based on the semantic understanding of the scene. This reasoning ability is based on semantic traversability, which is frequently achieved using semantic segmentation models fine-tuned on the testing domain. This fine-tuning process often involves manual data collection with the target robot and annotation by human labelers which is prohibitively expensive and unscalable. In this work, we present an effective methodology for training a semantic traversability estimator using egocentric videos and an automated annotation process. Egocentric videos are collected from a camera mounted on a pedestrian's chest. The dataset for training the semantic traversability estimator is then automatically generated by extracting semantically traversable regions in each video frame using a recent foundation model in image segmentation and its prompting technique. Extensive experiments with videos taken across several countries and cities, covering diverse urban scenarios, demonstrate the high scalability and generalizability of the proposed annotation method. Furthermore, performance analysis and real-world deployment for autonomous robot navigation showcase that the trained semantic traversability estimator is highly accurate, able to handle diverse camera viewpoints, computationally light, and real-world applicable.Keywords
All Related Versions
Funding Information
- Samsung Research Funding & Incubation Center for Future Technology at Samsung Electronics (SRFC-IT2002-02)
This publication has 25 references indexed in Scilit:
- Segment Anything2023
- Ego4D: Around the World in 3,000 Hours of Egocentric Video2022
- Masked-attention Mask Transformer for Universal Image Segmentation2022
- Visual Representation Learning for Preference-Aware Path Planning2022
- RELLIS-3D Dataset: Data, Benchmarks and Analysis2021
- Learning from Synthetic Data: Addressing Domain Shift for Semantic Segmentation2018
- Real-Time Semantic Mapping for Autonomous Off-Road Navigation2017
- Find your own way: Weakly-supervised segmentation of path proposals for urban autonomy2017
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015