RES-StS: Referring Expression Speaker via Self-Training With Scorer for Goal-Oriented Vision-Language Navigation
- 2 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (7) , 3441-3454
Abstract
It is a rather practical but difficult task to find a specified target object via autonomous exploration based on natural language descriptions in an unstructured environment. Since the human-annotated data is expensive to gather for the goal-oriented vision-language navigation (GVLN) task, the size of the standard dataset is inadequate, which has significantly limited the accuracy of previous techniques. In this work, we aim to improve the robustness and generalization of the navigator by dynamically providing high-quality pseudo-instructions using a proposed RES-StS paradigm. Specifically, we establish a referring expression speaker (RES) to predict descriptive instructions for the given path to the goal object. Based on an environment-and-object fusion (EOF) module, RES derives spatial representations from the input trajectories, which are subsequently encoded by a number of transformer layers. Additionally, given that the quality of the pseudo labels is important for data augmentation while the limited dataset may also hinder RES learning, we propose to equip RES with a more effective generation ability by using the self-training approach. A trajectory-instruction matching scorer (TIMS) network based on contrastive learning is proposed to selectively use rehearsal of prior knowledge. Finally, all network modules in the system are integrated by suggesting a multi-stage training strategy, allowing them to assist one another and thus enhance performance on the GVLN task. Experimental results demonstrate the effectiveness of our approach. Compared with the SOTA methods, our method improves SR, SPL, and RGS by 4.72%, 2.55%, and 3.45% respectively, on the REVERIE dataset, and 4.58%, 3.75% and 3.14% respectively, on the SOON dataset.Keywords
Funding Information
- National Natural Science Foundation of China (61733013, 62173248, 62233013)
- Shanghai Municipal Science and Technology Major Project (2021SHZDZX0100)
- Science and Technology Innovation Action Plan-AI Technology Support Project (22511104900)
- Fundamental Research Funds for the Central Universities
This publication has 47 references indexed in Scilit:
- Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout2019
- Multi-modal Discriminative Model for Vision-and-Language Navigation2019
- Deep Clustering for Unsupervised Learning of Visual Features2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments2018
- Matterport3D: Learning from RGB-D Data in Indoor Environments2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Deep Residual Learning for Image Recognition2016
- Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification2016
- CIDEr: Consensus-based image description evaluation2015