Towards Accurate and Smooth Human Motion Reconstruction from Sparse Tracking with State Space Model
Open Access
- 30 November 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Abstract
Reconstructing accurate full-body poses from sparse tracking is crucial in generative multimedia, yet achieving effective sequence modeling and high-quality generation remains challenging. Recently, the state space models (SSMs), especially Mamba, have showcased considerable promise in sequence modeling, indicating an appealing direction for building motion generation models. However, adapting SSM to this task is quite challenging due to two main aspects. Firstly, the sparsity of input conditions poses significant difficulty in accurately estimating full-body motions. Secondly, SSM lacks a specialized design to capture the temporal dependencies within motion sequences. To address these challenges, we propose a novel conditional diffusion model, Pose Mamba Diffusion (PMDiff), which comprises a bidirectional Pose Mamba Denoiser to achieve effective sequence modeling. Our PMDiff can effectively leverage the robust generation capabilities of diffusion models in conjunction with the sequence modeling proficiency of Mamba to track full-body poses accurately. Additionally, we notice that synthesized human motions often lack smoothness, resulting in unnatural motion sequences. To overcome this issue, we further introduce a learning-free Temporal Motion Filter (TMF), which can remarkably improve the smoothness of the generated human motion. Extensive experiments conducted on the large motion capture database (AMASS) show that our PMDiff outperforms previous state-of-the-art methods in terms of both accuracy and smoothness.Keywords
This publication has 34 references indexed in Scilit:
- Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model2023
- Generating Holistic 3D Human Motion from Speech2023
- MoFusion: A Framework for Denoising-Diffusion-Based Motion Synthesis2023
- AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- Swin Transformer V2: Scaling Up Capacity and Resolution2022
- Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors2022
- FLAG: Flow-based 3D Avatar Generation from Sparse Observations2022
- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows2022
- Expressive Body Capture: 3D Hands, Face, and Body From a Single Image2019