Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
Authors:
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2174-2181
Abstract
Learning meaningful visual representations in an embedding space can facilitate generalization in downstream tasks such as action segmentation and imitation. In this paper, we learn a motion-centric representation of surgical video demonstrations by grouping them into action segments/subgoals/options in a semi-supervised manner. We present Motion2Vec, an algorithm that learns a deep embedding feature space from video observations by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while pushed away from randomly sampled images of other segments, while respecting the temporal ordering of the images. The embeddings are iteratively segmented with a recurrent neural network for a given parametrization of the embedding space after pre-training the Siamese network. We only use a small set of labeled video segments to semantically align the embedding space and assign pseudo-labels to the remaining unlabeled data by inference on the learned model parameters. We demonstrate the use of this representation to imitate surgical suturing kinematic motions from publicly available videos of the JIGSAWS dataset. Results give 85.5% segmentation accuracy on average suggesting performance improvement over several state-of-the-art baselines, while kinematic pose imitation gives 0.94 centimeter error in position per observation on the test set. Videos, code and data are available at: https://sites.google.com/view/motion2vecKeywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Learning Robot Manipulation Tasks With Task-Parameterized Semitied Hidden Semi-Markov ModelIEEE Robotics and Automation Letters, 2016
- FaceNet: A unified embedding for face recognition and clustering2015
- An Improved Model for Segmentation and Recognition of Fine-Grained Activities with Application to Surgical Training Tasks2015
- Dynamical Movement Primitives: Learning Attractor Models for Motor BehaviorsNeural Computation, 2013
- Human-Machine Collaborative surgery using learned models2011
- Hidden semi-Markov modelsArtificial Intelligence, 2010
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Long Short-Term MemoryNeural Computation, 1997
- Robust part-of-speech tagging using a hidden Markov modelComputer Speech & Language, 1992
- A tutorial on hidden Markov models and selected applications in speech recognitionProceedings of the IEEE, 1989