Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model
Authors:
Preprint
Abstract
Current digital human studies focusing on lip-syncing and body movement are no longer sufficient to meet the growing industrial demand, while human video generation techniques that support interacting with real-world environments (e.g., objects) have not been well investigated. Despite human hand synthesis already being an intricate problem, generating objects in contact with hands and their interactions presents an even more challenging task, especially when the objects exhibit obvious variations in size and shape. To cope with these issues, we present a novel video Reenactment framework focusing on Human-Object Interaction (HOI) via an adaptive Layout-instructed Diffusion model (Re-HOLD). Our key insight is to employ specialized layout representation for hands and objects, respectively. Such representations enable effective disentanglement of hand modeling and object adaptation to diverse motion sequences. To further improve the generation quality of HOI, we have designed an interactive textural enhancement module for both hands and objects by introducing two independent memory banks. We also propose a layout-adjusting strategy for the cross-object reenactment scenario to adaptively adjust unreasonable layouts caused by diverse object sizes during inference. Comprehensive qualitative and quantitative evaluations demonstrate that our proposed framework significantly outperforms existing methods. Project page: https://fyycs.github.io/Re-HOLD.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- MoonShot: Towards Controllable Video Generation and Editing with Motion-Aware Multimodal ConditionsInternational Journal of Computer Vision, 2025
- Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance2024
- EMO: Emote Portrait Alive Generating Expressive Portrait Videos with Audio2Video Diffusion Model Under Weak Conditions2024
- Factorizing Text-to-Video Generation by Explicit Image Conditioning2024
- Cut-and-Paste: Subject-driven video editing with attention controlNeural Networks, 2024
- Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework2024
- Disco: Disentangled Control for Realistic Human Dance Generation2024
- DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion2023
- Thin-Plate Spline Motion Model for Image Animation2022
- VPU: A Video-Based Point Cloud Upsampling FrameworkIEEE Transactions on Image Processing, 2022