MMCPose: Multimodal Condition-Driven 3D Human Pose Estimation Via Diffusion Models
- 15 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28 (15209210) , 4373-4382
Abstract
Nowadays, diffusion-based methods for monocular 3D human pose estimation (3D HPE) have achieved state-of-the-art performance by directly regressing the 3D joint coordinates from the 2D observations. Although some methods incorporated the human body prior to improve the denoising quality, the absense of the structural relation and pose-aware guidance make these models prone to generating unreasonable poses. The challenge is noticeable in complex conditions such as occlusions and crowded scenarios. To alleviate this, we present MMCPose, a novel Multi-modal Condition-driven 3D HPE framework via diffusion models that capitalizes on the benefits of the multi-modal conditioning input. Specifically, we propose Multi-modal Condition Learning (MCL) strategy to incorporate multi-modal conditions such as joint- wise relation, part-aware prompt and pose-aware mask to improve the generation quality. The MCL block consists of (i) Joint- wise Relation Condition Learning (JRCL) models the flexible joint- wise relation via GCN to mitigate disturbances arising from confused joints. (ii) Part-aware Prompt Condition Learning (PPCL) constructs multi-granular prompts via accessible texts and feasible knowledge of body parts with learnable prompts to model implicit textual guidance. (iii) Pose-aware Mask Condition Learning (PMCL) designs a pose-specific mask to increase the model's emphasis to the pose region, augmenting the precision in capturing intricate pose details. Furthermore, we explore a multi-modal condition-pose interaction learning (MCPI) mechanism to establish interaction between the learned multi-modal conditions and poses to maximize the power of condition effect. This method fully unleashes the perceptual capability of the multi-modal conditions in diffusion-based 3D HPE. Extensive evaluations conducted on two popular benchmarks (e.g., Human3.6 M, MPI-INF-3DHP) and achieve new state-of-the-art performance.Keywords
Funding Information
- National Science and Technology Major Project
- Minister of Science and Technology, China (2021ZD0201403)
- Natural Science Foundation of Shanghai Municipality (24ZR1476900)
- Program of Shanghai Academic Research Leader (22XD1424500)
This publication has 36 references indexed in Scilit:
- DistilDoc: Knowledge Distillation for Visually-Rich Document Applications2024
- GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation2024
- SemiDocSeg: harnessing semi-supervised learning for document layout analysisInternational Journal on Document Analysis and Recognition (IJDAR), 2024
- P-STMO: Pre-trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Deep High-Resolution Representation Learning for Human Pose Estimation2019
- 3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training2019
- Cascaded Pyramid Network for Multi-person Pose Estimation2018
- Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision2017
- Stacked Hourglass Networks for Human Pose Estimation2016