Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction
Open Access
- 3 July 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (4) , 1-17
Abstract
We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency. Our method is fueled by a hybrid training dataset that consists of real-captured and rendered dynamic portrait videos with diverse subject appearances, facial motions, head poses, and known lighting conditions. Specifically, we construct an LED-based lighting system for realistic lighting emulation and high-speed video relighting data acquisition. By leveraging the image priors embedded in pre-trained video diffusion models, and using per-frame high dynamic range (HDR) environment map as lighting control, we train a high-performance generative model for realistic and identity-preserving dynamic portrait video relighting. In addition to the environment map control, our model uses a synthesized background image to enable control on the camera's exposure level and color tone. Our model can produce temporally consistent relit portrait video that looks realistic and harmonious under a provided new environment and faithfully preserve the subject's expression and fine facial features, including skin tone, wrinkles, and facial hair. Our model generalizes well to unseen data, in terms of the subject appearance, motion, and lighting condition. We perform extensive experiments on relighting in-the-wild videos with various environment maps and demonstrate practical applications on portrait photography. Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency. Our project page: https://yufanzhang82.github.io/PixelCube/.Keywords
All Related Versions
This publication has 40 references indexed in Scilit:
- Relightable Gaussian Codec Avatars2024
- Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation2024
- Relightful Harmonization: Lighting-Aware Portrait Background Replacement2024
- Real-Time 3D-Aware Portrait Video Relighting2024
- FaceLit: Neural 3D Relightable Faces2023
- SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model2023
- Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT DatasetInternational Journal of Computer Vision, 2023
- Face Relighting with Geometrically Consistent Shadows2022
- Estimation of continuous valence and arousal levels from faces in naturalistic conditionsNature Machine Intelligence, 2021
- Intrinsic Face Image Decomposition with Human Face Priors2014