Affordances from Human Videos as a Versatile Representation for Robotics
Authors:
- Shikhar BahlSBShikhar Bahl
- Russell MendoncaRMRussell Mendonca
- Lili ChenLCLili Chen
- Unnat JainUJUnnat Jain
- Deepak PathakDPDeepak Pathak
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 01-13
Abstract
Building a robot that can understand and learn to interact by watching humans has inspired several vision problems. However, despite some successful results on static datasets, it remains unclear how current models can be used on a robot directly. In this paper, we aim to bridge this gap by leveraging videos of human interactions in an environment centric manner. Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show how to seamlessly integrate our affordance model with four robot learning paradigms including offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. We show the efficacy of our approach, which we call Vision-Robotics Bridge (VRB) across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild.Keywords
All Related Versions
This publication has 67 references indexed in Scilit:
- Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture2016
- Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours2016
- Probabilistic movement primitives for coordination of multiple human–robot collaborative tasksAutonomous Robots, 2016
- Learning object affordances by leveraging the combination of human-guidance and self-exploration2016
- Affordance detection of tool parts from geometric features2015
- Interactive Hierarchical Task Learning from a Single Demonstration2015
- Human-level control through deep reinforcement learningNature, 2015
- Dynamic Movement Primitives for Human-Robot interaction: Comparison with human behavioral observation2013
- Scene Parsing by Integrating Function, Geometry and Appearance Models2013
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009