Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model
Authors:
Preprint
Abstract
Current digital human studies focusing on lip-syncing and body movement are no longer sufficient to meet the growing industrial demand, while human video generation techniques that support interacting with real-world environments (e.g., objects) have not been well investigated. Despite human hand synthesis already being an intricate problem, generating objects in contact with hands and their interactions presents an even more challenging task, especially when the objects exhibit obvious variations in size and shape. To tackle these issues, we present a novel video Reenactment framework focusing on Human-Object Interaction (HOI) via an adaptive Layout-instructed Diffusion model (Re-HOLD). Our key insight is to employ specialized layout representation for hands and objects, respectively. Such representations enable effective disentanglement of hand modeling and object adaptation to diverse motion sequences. To further improve the generation quality of HOI, we design an interactive textural enhancement module for both hands and objects by introducing two independent memory banks. We also propose a layout adjustment strategy for the cross-object reenactment scenario to adaptively adjust unreasonable layouts caused by diverse object sizes during inference. Comprehensive qualitative and quantitative evaluations demonstrate that our proposed framework significantly outperforms existing methods. Project page: https://fyycs.github.io/Re-HOLD.Keywords
All Related Versions
This publication has 50 references indexed in Scilit:
- MoonShot: Towards Controllable Video Generation and Editing with Motion-Aware Multimodal ConditionsInternational Journal of Computer Vision, 2025
- Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance2024
- Iterative Ensemble Training with Anti-gradient Control for Mitigating Memorization in Diffusion Models2024
- EMO: Emote Portrait Alive Generating Expressive Portrait Videos with Audio2Video Diffusion Model Under Weak Conditions2024
- Factorizing Text-to-Video Generation by Explicit Image Conditioning2024
- GraspXL: Generating Grasping Motions for Diverse Objects at Scale2024
- LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning2024
- Cut-and-Paste: Subject-driven video editing with attention controlNeural Networks, 2024
- PLACE: Proximity Learning of Articulation and Contact in 3D Environments2020
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015