Self-Supervised Learning of Action Affordances as Interaction Modes
Authors:
- Liquan WangLWLiquan Wang
- Nikita DvornikNDNikita Dvornik
- Rafael DubeauRDRafael Dubeau
- Mayank MittalMMMayank Mittal
- Animesh GargAGAnimesh Garg
Preprint
Abstract
When humans perform a task with an articulated object, they interact with the object only in a handful of ways, while the space of all possible interactions is nearly endless. This is because humans have prior knowledge about what interactions are likely to be successful, i.e., to open a new door we first try the handle. While learning such priors without supervision is easy for humans, it is notoriously hard for machines. In this work, we tackle unsupervised learning of priors of useful interactions with articulated objects, which we call interaction modes. In contrast to the prior art, we use no supervision or privileged information; we only assume access to the depth sensor in the simulator to learn the interaction modes. More precisely, we define a successful interaction as the one changing the visual environment substantially and learn a generative model of such interactions, that can be conditioned on the desired goal state of the object. In our experiments, we show that our model covers most of the human interaction modes, outperforms existing state-of-the-art methods for affordance learning, and can generalize to objects never seen during training. Additionally, we show promising results in the goal-conditional setup, where our model can be quickly fine-tuned to perform a given task. We show in the experiments that such affordance learning predicts interaction which covers most modes of interaction for the querying articulated object and can be fine-tuned to a goal-conditional model. For supplementary: https://actaim.github.io.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation2022
- KPAM: KeyPoint Affordances for Category-Level Robotic Manipulation2022
- Universal Manipulation Policy Network for Articulated ObjectsIEEE Robotics and Automation Letters, 2022
- Where2Act: From Pixels to Actions for Articulated 3D Objects2021
- ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory2021
- Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes2021
- Learning Hybrid Object Kinematics for Efficient Hierarchical Planning Under Uncertainty2020
- KETO: Learning Keypoint Representations for Tool Manipulation2020
- 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions2017
- Visual object-action recognition: Inferring object affordances from human demonstrationComputer Vision and Image Understanding, 2011