Learning to Follow and Generate Instructions for Language-Capable Navigation
- 13 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (5) , 3334-3350
Abstract
Visual-language navigation (VLN) is a challenging task that requires embodied agents to follow natural language instructions to navigate in previously unseen environments. However, existing literature put most emphasis on interpreting instructions into actions, only delivering “dumb” wayfinding agents which cannot actively use natural language to communicate with humans. In this article, we devise Lana , a language-capable n avigation a gent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively, for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We further extend Lana by exploiting object semantics during route encoding. This leads to Lana+ , a more powerful framework that simulates the way humans refer to landmarks for instructions composition and wayfinding. We empirically verify that, compared with recent advanced task-specific solutions, Lana attains better performances on both instruction following and generation, with nearly half complexity. In addition, endowed with language generation capability, Lana can explain to humans its behaviors and assist human's wayfinding. Benefiting from landmark information, Lana+ exhibits even more impressive performance. This work is expected to foster future efforts towards building more trustworthy and socially-intelligent navigation robots.Keywords
Funding Information
- National Natural Science Foundation of China (62293554)
- Fundamental Research Funds for the Central Universities (226-2022-00051)
- CCF-Tencent Open Fund
This publication has 86 references indexed in Scilit:
- ASSISTER: Assistive Navigation via Conditional Instruction Generation2022
- Learning from Unlabeled 3D Environments for Vision-and-Language Navigation2022
- Environment-Agnostic Multitask Learning for Natural Language Grounded Navigation2020
- Active Visual Information Gathering for Vision-Language Navigation2020
- Improving Vision-and-Language Navigation with Image-Text Pairs from the Web2020
- Object-and-Action Aware Model for Visual Language Navigation2020
- UNITER: UNiversal Image-TExt Representation Learning2020
- Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation2018
- Elements of Good Route Directions in Familiar and Unfamiliar Environments1999
- From knowledge to words to wayfinding: Issues in the production and comprehension of route directions1997