DyCrowd: Towards Dynamic Crowd Reconstruction from a Large-scene Video
Preprint
Abstract
3D reconstruction of dynamic crowds in large scenes has become increasingly important for applications such as city surveillance and crowd analysis. However, current works attempt to reconstruct 3D crowds from a static image, causing a lack of temporal consistency and inability to alleviate the typical impact caused by occlusions. In this paper, we propose DyCrowd, the first framework for spatio-temporally consistent 3D reconstruction of hundreds of individuals' poses, positions and shapes from a large-scene video. We design a coarse-to-fine group-guided motion optimization strategy for occlusion-robust crowd reconstruction in large scenes. To address temporal instability and severe occlusions, we further incorporate a VAE (Variational Autoencoder)-based human motion prior along with a segment-level group-guided optimization. The core of our strategy leverages collective crowd behavior to address long-term dynamic occlusions. By jointly optimizing the motion sequences of individuals with similar motion segments and combining this with the proposed Asynchronous Motion Consistency (AMC) loss, we enable high-quality unoccluded motion segments to guide the motion recovery of occluded ones, ensuring robust and plausible motion recovery even in the presence of temporal desynchronization and rhythmic inconsistencies. Additionally, in order to fill the gap of no existing well-annotated large-scene video dataset, we contribute a virtual benchmark dataset, VirtualCrowd, for evaluating dynamic crowd reconstruction from large-scene videos. Experimental results demonstrate that the proposed method achieves state-of-the-art performance in the large-scene dynamic crowd reconstruction task. The code and dataset will be available for research purposes.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- TRAM: Global Trajectory and Motion of 3D Humans from in-the-Wild Videos2024
- Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras With Human SemanticsIEEE Transactions on Circuits and Systems for Video Technology, 2023
- SMPL: A Skinned Multi-Person Linear Model2023
- MILI: Multi-person inference from a low-resolution imageFundamental Research, 2023
- PVDet: Towards pedestrian and vehicle detection on gigapixel-level imagesEngineering Applications of Artificial Intelligence, 2022
- 3D Human Pose, Shape and Texture from Low-Resolution Images and VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Collective Density Clustering for Coherent Motion DetectionIEEE Transactions on Multimedia, 2017
- Multiscale gigapixel photographyNature, 2012
- Clustering by Passing Messages Between Data PointsScience, 2007
- Dynamic programming algorithm optimization for spoken word recognitionIEEE Transactions on Acoustics, Speech, and Signal Processing, 1978