A Language-Driven Navigation Strategy Integrating Semantic Maps and Large Language Models
Authors:
- 14 October 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9753-9760
Abstract
Accurate perception of semantic and spatial information is crucial for robots performing language-driven navigation tasks. Existing approaches utilize visual-language models to extract semantic information from the environment and construct maps. However, constrained by the generalization and accuracy of these models themselves, the constructed maps may not be accurate and comprehensive, thereby affecting the accuracy of navigation tasks. Inspired by foundational models’ outstanding classification and segmentation capabilities, this study introduces a semantic map constructed using foundational models. We leverage a foundational model to semantically segment objects in the robot’s video stream and fuse semantics onto the map. Furthermore, this map is used in conjunction with large language models (LLMs) that receive natural language instructions to complete the navigation task. A substantial number of experiments in a simulated environment demonstrate that our method outperforms existing ones in language-driven navigation tasks.Keywords
Funding Information
- National Natural Science Foundation of China
This publication has 32 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning2022
- Cross-modal Map Learning for Vision and Language Navigation2022
- SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences2021
- Topological Planning with Transformers for Vision-and-Language Navigation2021
- Learning to Plan with Uncertain Topological Maps2020
- Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments2020
- Occupancy Anticipation for Efficient Exploration and Navigation2020
- Matterport3D: Learning from RGB-D Data in Indoor Environments2017
- SLAM++: Simultaneous Localisation and Mapping at the Level of Objects2013