Physical Representation Learning and Parameter Identification from Video Using Differentiable Physics
Open Access
- 17 October 2021
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 130 (1) , 3-16
Abstract
Representation learning for video is increasingly gaining attention in the field of computer vision. For instance, video prediction models enable activity and scene forecasting or vision-based planning and control. In this article, we investigate the combination of differentiable physics and spatial transformers in a deep action conditional video representation network. By this combination our model learns a physically interpretable latent representation and can identify physical parameters. We propose supervised and self-supervised learning methods for our architecture. In experiments, we consider simulated scenarios with pushing, sliding and colliding objects, for which we also analyze the observability of the physical properties. We demonstrate that our network can learn to encode images and identify physical properties like mass and friction from videos and action sequences. We evaluate the accuracy of our training methods, and demonstrate the ability of our method to predict future video frames from input images and actions.Keywords
Funding Information
- Cyber Valley
- Max-Planck-Gesellschaft
- BMBF (01IS18039B)
This publication has 13 references indexed in Scilit:
- Learning to Disentangle Latent Physical Factors for Video PredictionLecture Notes in Computer Science, 2019
- A Differentiable Physics Engine for Deep Learning in RoboticsFrontiers in Neurorobotics, 2019
- Interpretable Intuitive Physics Model2018
- Deep visual foresight for planning robot motion2017
- “What Happens If...” Learning to Predict the Effect of Forces in Images2016
- Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images2016
- CVXGEN: a code generator for embedded convex optimizationOptimization and Engineering, 2011
- Rigid-Body Dynamics with Friction and ImpactSIAM Review, 2000
- Long Short-Term MemoryNeural Computation, 1997
- Formulating Dynamic Multi-Rigid-Body Contact Problems with Friction as Solvable Linear Complementarity ProblemsNonlinear Dynamics, 1997