Building Generalizable Agents with a Realistic and Rich 3D Environment
Preprint
Abstract
Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et.al.). The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel- & task-level augmentations such as domain randomization (Toubin et. al.) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at http://github.com/facebookresearch/House3D.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Cognitive Mapping and Planning for Visual NavigationInternational Journal of Computer Vision, 2019
- Gated-Attention Architectures for Task-Oriented Language GroundingProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning2017
- Domain randomization for transferring deep neural networks from simulation to the real world2017
- CAD2RL: Real Single-Image Flight Without a Single Real Image2017
- Curiosity-Driven Exploration by Self-Supervised Prediction2017
- Semantic Scene Completion from a Single Depth Image2017
- Gated-Attention Readers for Text Comprehension2017
- Directed Sonar Sensing for Mobile Robot Navigation1992