Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10996-11006
Abstract
Multi-frame human pose estimation has long been a compelling and fundamental problem in computer vision. This task is challenging due to fast motion and pose occlusion that frequently occur in videos. State-of-the-art methods strive to incorporate additional visual evidences from neighboring frames (supporting frames) to facilitate the pose estimation of the current frame (key frame). One aspect that has been obviated so far, is the fact that current methods directly aggregate unaligned contexts across frames. The spatial-misalignment between pose features of the current frame and neighboring frames might lead to unsatisfactory results. More importantly, existing approaches build upon the straightforward pose estimation loss, which unfortunately cannot constrain the network to fully leverage useful information from neighboring frames. To tackle these problems, we present a novel hierarchical alignment framework, which leverages coarse-to-fine deformations to progressively update a neighboring frame to align with the current frame at the feature level. We further propose to explicitly supervise the knowledge extraction from neighboring frames, guaranteeing that useful complementary cues are extracted. To achieve this goal, we theoretically analyzed the mutual information between the frames and arrived at a loss that maximizes the task-relevant mutual information. These allow us to rank No.1 in the Multi-frame Person Pose Estimation Challenge on benchmark dataset PoseTrack2017, and obtain state-of-the-art performance on benchmarks Sub-JHMDB and Pose-Track2018. Our code is released at https://github.com/Pose-Group/FAMI-Pose, hoping that it will be useful to the community.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61902348)
This publication has 50 references indexed in Scilit:
- Convolutional Pose Machines2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- Flowing ConvNets for Human Pose Estimation in Videos2015
- Joint action recognition and pose estimation from video2015
- DeepPose: Human Pose Estimation via Deep Neural Networks2014
- Towards Understanding Action Recognition2013
- Beyond Physical Connections: Tree Models in Human Pose Estimation2013
- N-best maximal decoders for part models2011
- Cascaded Models for Articulated Pose Estimation2010
- Multiple Tree Models for Occlusion and Spatial Constraints in Human Pose Estimation2008