FTCM: Frequency-Temporal Collaborative Module for Efficient 3D Human Pose Estimation in Video
- 21 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (2) , 911-923
Abstract
Capturing cross-pose correlation from a sequence of frame-level 2D poses is essential for 3D human pose estimation (3D-HPE) in the video. Recent studies have shown the promising potential of modeling the pose relation with feature-mixing operations on the temporal domain. However, they seldom consider the interaction across poses in the frequency domain. This paper studies a Frequency-Temporal Collaborative Module (FTCM) to explore the feasibility of encoding the cross-pose correlations in both frequency and temporal domains. FTCM aims to jointly capture the global and local cross-pose correlations with a more lightweight network model. Specifically, FTCM splits the pose features into two groups along the channel dimension and separately models the frequency and temporal interactions across poses with different feature-mixing operations in parallel. To achieve this goal, we purposely design two pose-mixing units, i.e., the frequency pose-mixing (FPM) and the temporal pose-mixing (TPM). Particularly, FPM is designed to reap the global correlations among different pose frequencies with the representation obtained by converting the original pose signals with Fast Fourier transform (FFT). Unlike the pose-mixing used by previous methods like Transformers that influences an individual pose with all other poses, TPM locally calibrates the pose with dynamics aggregated within several adjacent poses in the temporal domain, explicitly weighting neighboring poses more with respect to the far-away ones so as to enforce a strict locality constraint. Besides, the group strategy significantly reduces the model complexity. To verify the effectiveness of FTCM, we conduct extensive experiments on two benchmarks (i.e., Human3.6M and MPI-INF-3DHP). Experimental results not only exhibit favorable accuracy/complexity trade-offs of our FTCM but also show superior or comparable performance to state-of-the-art methods on both datasets. The code and model are publicly available at: https://github.com/zhenhuat/FTCM.Keywords
Funding Information
- National Natural Science Foundation of China (61932009, 62202139)
- Anhui Provincial Natural Science Foundation (2208085QF191)
This publication has 55 references indexed in Scilit:
- A Simple Yet Effective Baseline for 3d Human Pose Estimation2017
- Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose2017
- Structured Prediction of 3D Human Pose with Deep Neural Networks2016
- Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural EnvironmentsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Latent structured models for human pose estimation2011
- Shape Recognition and Pose Estimation for Mobile Augmented RealityIEEE Transactions on Visualization and Computer Graphics, 2010
- Pose estimation and adaptive robot behaviour for human-robot interaction2009
- Recovering 3D human pose from monocular imagesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2005
- Real-Time Vision for Human-Computer Interaction2005
- Gauss and the history of the fast fourier transformIEEE ASSP Magazine, 1984