Understanding Action Sequences based on Video Captioning for Learning-from-Observation
Authors:
- Iori YanokuraIYIori Yanokura
- Naoki WakeNWNaoki Wake
- Kazuhiro SasabuchiKSKazuhiro Sasabuchi
- Katsushi IkeuchiKIKatsushi Ikeuchi
- Masayuki InabaMIMasayuki Inaba
Preprint
Abstract
Learning actions from human demonstration video is promising for intelligent robotic systems. Extracting the exact section and re-observing the extracted video section in detail is important for imitating complex skills because human motions give valuable hints for robots. However, the general video understanding methods focus more on the understanding of the full frame,lacking consideration on extracting accurate sections and aligning them with the human's intent. We propose a Learning-from-Observation framework that splits and understands a video of a human demonstration with verbal instructions to extract accurate action sequences. The splitting is done based on local minimum points of the hand velocity, which align human daily-life actions with object-centered face contact transitions required for generating robot motion. Then, we extract a motion description on the split videos using video captioning techniques that are trained from our new daily-life action video dataset. Finally, we match the motion descriptions with the verbal instructions to understand the correct human intent and ignore the unintended actions inside the video. We evaluate the validity of hand velocity-based video splitting and demonstrate that it is effective. The experimental results on our new video captioning dataset focusing on daily-life human actions demonstrate the effectiveness of the proposed method. The source code, trained models, and the dataset will be made available.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networksRobotics and Autonomous Systems, 2018
- BSN: Boundary Sensitive Network for Temporal Action Proposal Generation2018
- Describing Upper-Body Motions Based on Labanotation for Learning-from-Observation RobotsInternational Journal of Computer Vision, 2018
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs2016
- Change-point detection in time-series data by relative density-ratio estimationNeural Networks, 2013
- Sequential change‐point detection based on direct density‐ratio estimationStatistical Analysis and Data Mining: An ASA Data Science Journal, 2011
- Incremental kinesthetic teaching of motion primitives using the motion refinement tubeAutonomous Robots, 2011
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Detecting dance motion structure through music analysis2004