Learn to Predict How Humans Manipulate Large-Sized Objects From Interactive Motions
- 14 February 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (2) , 4702-4709
Abstract
Understanding human intentions during interactions has been a long-lasting theme, that has applications in human-robot interaction, virtual reality and surveillance. In this study, we focus on full-body human interactions with large-sized daily objects and aim to predict the future states of objects and humans given a sequential observation of human-object interaction. As there is no such dataset dedicated to full-body human interactions with large-sized daily objects, we collected a large-scale dataset containing thousands of interactions for training and evaluation purposes. We also observe that an object’s intrinsic physical properties are useful for the object motion prediction, and thus design a set of object dynamic descriptors to encode such intrinsic properties. We treat the object dynamic descriptors as a new modality and propose a graph neural network, HO-GCN, to fuse motion data and dynamic descriptors for the prediction task. We show the proposed network that consumes dynamic descriptors can achieve state-of-the-art prediction results and help the network better generalize to unseen objects. We also demonstrate the predicted results are useful for human-robot collaborations.Keywords
All Related Versions
Funding Information
- ERC Consolidator
- 4DRepLy (770784)
- Lise Meitner Postdoctoral Fellowship
- Innovation and Technology Commission of the HKSAR Goverment
- InnoHK
This publication has 33 references indexed in Scilit:
- Investigation of different skeleton features for CNN-based 3D action recognition2017
- Malleable Embodiment2017
- A system for learning continuous human-robot interactions from human-human demonstrations2017
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- Skeleton based action recognition with convolutional neural network2015
- The KIT whole-body human motion database2015
- An Integrated Framework for Human–Robot Collaborative ManipulationIEEE Transactions on Cybernetics, 2014
- Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural EnvironmentsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Berkeley MHAD: A comprehensive Multimodal Human Action Database2013
- View invariant human action recognition using histograms of 3D joints2012