Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
Preprint
Abstract
Vision-Language Navigation (VLN) is a task where agents learn to navigate following natural language instructions. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional approaches exploit the vision and language features in cross-modal grounding. However, the VLN task remains challenging, since previous works have neglected the rich semantic information contained in the environment (such as implicit navigation graphs or sub-trajectory semantics). In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information. The auxiliary tasks have four reasoning objectives: explaining the previous actions, estimating the navigation progress, predicting the next orientation, and evaluating the trajectory consistency. As a result, these additional training signals help the agent to acquire knowledge of semantic representations in order to reason about its activity and build a thorough perception of the environment. Our experiments indicate that auxiliary reasoning tasks improve both the performance of the main task and the model generalizability by a large margin. Empirically, we demonstrate that an agent trained with self-supervised auxiliary reasoning tasks substantially outperforms the previous state-of-the-art method, being the best existing approach on the standard benchmark.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Cognitive Mapping and Planning for Visual NavigationInternational Journal of Computer Vision, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation2018
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- ViZDoom: A Doom-based AI research platform for visual reinforcement learning2016
- Deep Residual Learning for Image Recognition2016
- Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference.The Journal of Philosophy, 1991