Pose-based Contrastive Learning for Domain Agnostic Activity Representations
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3432-3442
Abstract
While recognition accuracies of video classification models trained on conventional benchmarks are gradually saturating, recent studies raise alarm about the learned representations not generalizing well across different domains. Learning abstract concepts behind an activity instead of overfitting to the appearances and biases of a specific benchmark domain is vital for building generalizable behaviour understanding models. In this paper, we introduce Pose-based High Level View Contrasting (P-HLVC), a novel method that leverages human pose dynamics as a supervision signal aimed at learning domain-invariant activity representations. Our model learns to link image sequences to more abstract body pose information through iterative contrastive clustering and the Sinkhorn-Knopp algorithm, providing us with video representations more resistant to domain shifts. We demonstrate the effectiveness of our approach in a cross-domain action recognition setting and achieve significant improvements on the synthetic-to-real Sims4Action benchmark.1Keywords
This publication has 34 references indexed in Scilit:
- PoTion: Pose MoTion Representation for Action Recognition2018
- VirtualHome: Simulating Household Activities Via Programs2018
- Unsupervised Representation Learning by Sorting Sequences2017
- RMPE: Regional Multi-person Pose Estimation2017
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization2017
- View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data2017
- Procedural Generation of Videos to Train Deep Action Recognition Networks2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Unsupervised Visual Representation Learning by Context Prediction2015
- HMDB: A large video database for human motion recognition2011