Learning Human Search Behavior from Egocentric Visual Inputs
Preprint
Abstract
"Looking for things" is a mundane but critical task we repeatedly carry on in our daily life. We introduce a method to develop a human character capable of searching for a randomly located target object in a detailed 3D scene using its locomotion capability and egocentric vision perception represented as RGBD images. By depriving the privileged 3D information from the human character, it is forced to move and look around simultaneously to account for the restricted sensing capability, resulting in natural navigation and search behaviors. Our method consists of two components: 1) a search control policy based on an abstract character model, and 2) an online replanning control module for synthesizing detailed kinematic motion based on the trajectories planned by the search policy. We demonstrate that the combined techniques enable the character to effectively find often occluded household items in indoor environments. The same search policy can be applied to different full-body characters without the need for retraining. We evaluate our method quantitatively by testing it on randomly generated scenarios. Our work is a first step toward creating intelligent virtual agents with humanlike behaviors driven by onboard sensors, paving the road toward future robotic applications.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation2020
- Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered EnvironmentsIEEE Robotics and Automation Letters, 2020
- Learning character-agnostic motion for motion retargeting in 2DACM Transactions on Graphics, 2019
- Mode-adaptive neural networks for quadruped motion controlACM Transactions on Graphics, 2018
- Deep learning of biomimetic sensorimotor control for biomechanical human animationACM Transactions on Graphics, 2018
- Phase-functioned neural networks for character controlACM Transactions on Graphics, 2017
- Target-driven visual navigation in indoor scenes using deep reinforcement learning2017
- A deep learning framework for character motion synthesis and editingACM Transactions on Graphics, 2016
- EyecatchACM Transactions on Graphics, 2012