Name
Title | ||
|---|---|---|
Reinforced Cross-Modal Matching And Self-Supervised Imitation Learning For Vision-Language Navigation |
Abstract | ||
|---|---|---|
Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and tV rajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%). |
Year | DOI | Venue |
|---|---|---|
2018 | 10.1109/CVPR.2019.00679 | 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019) |
Field | DocType | Volume |
Generalizability theory,Global matching,Computer science,Embodied agent,Natural language,Artificial intelligence,Imitation learning,Performance gap,Machine learning,Modal,Reinforcement learning | Journal | abs/1811.10092 |
ISSN | Citations | PageRank |
1063-6919 | 14 | 0.52 |
References | Authors | |
30 | 8 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Xin Wang | 1 | 110 | 13.59 |
| Qiuyuan Huang | 2 | 176 | 17.66 |
| Asli Çelikyilmaz | 3 | 407 | 39.06 |
| Jianfeng Gao | 4 | 5729 | 296.43 |
| Dinghan Shen | 5 | 108 | 10.37 |
| Yuan-Fang Wang | 6 | 835 | 137.72 |
| William Yang Wang | 7 | 493 | 59.64 |
| Lei Zhang | 8 | 2533 | 164.29 |