Yu Ji
Yu Ji has a total of 17 co-authors. Here are the top co-authors by number of shared publications:
- Jinwei Ye
George Mason University, Fairfax, USA
5 shared publications - Changxi Zheng
Columbia University, New York, USA
3 shared publications - Yu Guo
George Mason University
3 shared publications - Yanchen Liu
Columbia University
2 shared publications - Yufan Zhang
George Mason University, Fairfax, USA
2 shared publications - Yuqi Ding
Division of Computer Science and Engineering, Louisiana State University, Baton Rouge, LA, USA
2 shared publications
Latest Publications
- Open Access
- 3 July 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (4) , 1-17
Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction
We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency. Our method is fueled by a hybrid training dataset that consists of real-captured and rendered dynamic portrait videos with diverse subject appearances, facial motions, head poses, and known lighting conditions. Specifically, we construct an LED-based lighting system for realistic lighting emulation and high-speed video relighting data acquisition. By leveraging the image priors embedded in pre-trained video diffusion models, and using per-frame high dynamic range (HDR) environment map as lighting control, we train a high-performance generative model for realistic and identity-preserving dynamic portrait video relighting. In addition to the environment map control, our model uses a synthesized background image to enable control on the camera's exposure level and color tone. Our model can produce temporally consistent relit portrait video that looks realistic and harmonious under a provided new environment and faithfully preserve the subject's expression and fine facial features, including skin tone, wrinkles, and facial hair. Our model generalizes well to unseen data, in terms of the subject appearance, motion, and lighting condition. We perform extensive experiments on relighting in-the-wild videos with various environment maps and demonstrate practical applications on portrait photography. Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency. Our project page: https://yufanzhang82.github.io/PixelCube/.
- 3 July 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (4) , 1-17
Open AccessPixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction
We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency. Our method is fueled by a hybrid training dataset that consists of real-captured and rendered dynamic portrait videos with diverse subject appearances, facial motions, head poses, and known lighting conditions. Specifically, we construct an LED-based lighting system for realistic lighting emulation and high-speed video relighting data acquisition. By leveraging the image priors embedded in pre-trained video diffusion models, and using per-frame high dynamic range (HDR) environment map as lighting control, we train a high-performance generative model for realistic and identity-preserving dynamic portrait video relighting. In addition to the environment map control, our model uses a synthesized background image to enable control on the camera's exposure level and color tone. Our model can produce temporally consistent relit portrait video that looks realistic and harmonious under a provided new environment and faithfully preserve the subject's expression and fine facial features, including skin tone, wrinkles, and facial hair. Our model generalizes well to unseen data, in terms of the subject appearance, motion, and lighting condition. We perform extensive experiments on relighting in-the-wild videos with various environment maps and demonstrate practical applications on portrait photography. Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency. Our project page: https://yufanzhang82.github.io/PixelCube/.
- 29 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-14
Snapshot 3D Gaussian Splatting for Miniature Scenes
We present a snapshot imaging technique for recovering 3D surrounding views of miniature scenes. Due to their intricacy, miniature scenes with objects sized in millimeters are difficult to reconstruct. Yet miniatures are common in life and their 3D digitalization is desirable. We design a catadioptric imaging system with a single camera and multiple pairs of planar mirrors for snapshot 3D reconstruction of miniature scenes from a dollhouse perspective. We first present an in-depth analysis on how to configure catadioptric imaging systems that use a pair of planar mirrors. We derive mirror parameters (e.g., orientation and position) by solving a viewpoint mapping problem, which aims to determine viable mirror configuration that is able to provide reflection image from a desired viewpoint. By applying the design principle, We show a full-surround snapshot catadioptric imaging system built with eight pairs of planar mirrors. Specifically, we place the mirror pairs on nested pyramid surfaces with different angles for capturing surrounding multi-view images in a single shot. This would allow 3D reconstruction of dynamic scenes. Our mirror design is customizable based on the size of the scene for optimized view coverage. We use the 3D Gaussian Splatting (3DGS) representation for scene reconstruction and novel view synthesis. We overcome the challenge posed by our sparse view input by integrating visual hull-derived depth constraint. We perform experiments on rendered synthetic images and real images of a variety of miniature scenes that are captured by our custom-built imaging system. Experimental results demonstrate that our method outperforms state-of-the-art sparse-view and calibration-free 3DGS methods. We also show novel view synthesis results on dynamic miniature scenes (e.g., a moving live insect).
- 29 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-14
Snapshot 3D Gaussian Splatting for Miniature Scenes
We present a snapshot imaging technique for recovering 3D surrounding views of miniature scenes. Due to their intricacy, miniature scenes with objects sized in millimeters are difficult to reconstruct. Yet miniatures are common in life and their 3D digitalization is desirable. We design a catadioptric imaging system with a single camera and multiple pairs of planar mirrors for snapshot 3D reconstruction of miniature scenes from a dollhouse perspective. We first present an in-depth analysis on how to configure catadioptric imaging systems that use a pair of planar mirrors. We derive mirror parameters (e.g., orientation and position) by solving a viewpoint mapping problem, which aims to determine viable mirror configuration that is able to provide reflection image from a desired viewpoint. By applying the design principle, We show a full-surround snapshot catadioptric imaging system built with eight pairs of planar mirrors. Specifically, we place the mirror pairs on nested pyramid surfaces with different angles for capturing surrounding multi-view images in a single shot. This would allow 3D reconstruction of dynamic scenes. Our mirror design is customizable based on the size of the scene for optimized view coverage. We use the 3D Gaussian Splatting (3DGS) representation for scene reconstruction and novel view synthesis. We overcome the challenge posed by our sparse view input by integrating visual hull-derived depth constraint. We perform experiments on rendered synthetic images and real images of a variety of miniature scenes that are captured by our custom-built imaging system. Experimental results demonstrate that our method outperforms state-of-the-art sparse-view and calibration-free 3DGS methods. We also show novel view synthesis results on dynamic miniature scenes (e.g., a moving live insect).
- 18 September 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (9) , 7243-7254
Textureless Deformable Object Tracking With Invisible Markers
Tracking and reconstructing deformable objects with little texture is challenging due to the lack of features. Here we introduce “invisible markers” for accurate and robust correspondence matching and tracking. Our markers are visible only under ultraviolet (UV) light. We build a novel imaging system for capturing videos of deformed objects under their original untouched appearance (which may have little texture) and, simultaneously, with our markers. We develop an algorithm that first establishes accurate correspondences using video frames with markers, and then transfers them to the untouched views as ground-truth labels. In this way, we are able to generate high-quality labeled data for training learning-based algorithms. We contribute a large real-world dataset, DOT, for tracking deformable objects with little or no texture. Our dataset has about one million video frames of various types of deformable objects. We provide ground truth tracked correspondences in both 2D and 3D. We benchmark state-of-the-art methods on optical flow and deformable object reconstruction using our dataset, which poses great challenges. By training on DOT, their performance significantly improves, not only on our dataset, but also on other unseen data.
- 18 September 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (9) , 7243-7254
Textureless Deformable Object Tracking With Invisible Markers
Tracking and reconstructing deformable objects with little texture is challenging due to the lack of features. Here we introduce “invisible markers” for accurate and robust correspondence matching and tracking. Our markers are visible only under ultraviolet (UV) light. We build a novel imaging system for capturing videos of deformed objects under their original untouched appearance (which may have little texture) and, simultaneously, with our markers. We develop an algorithm that first establishes accurate correspondences using video frames with markers, and then transfers them to the untouched views as ground-truth labels. In this way, we are able to generate high-quality labeled data for training learning-based algorithms. We contribute a large real-world dataset, DOT, for tracking deformable objects with little or no texture. Our dataset has about one million video frames of various types of deformable objects. We provide ground truth tracked correspondences in both 2D and 3D. We benchmark state-of-the-art methods on optical flow and deformable object reconstruction using our dataset, which poses great challenges. By training on DOT, their performance significantly improves, not only on our dataset, but also on other unseen data.
- 25 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (7) , 4597-4611
Polarimetric Helmholtz Stereopsis
Helmholtz stereopsis (HS) exploits the reciprocity principle of light propagation (i.e., the Helmholtz reciprocity) for 3D reconstruction of surfaces with arbitrary reflectance. In this paper, we present the polarimetric Helmholtz stereopsis (polar-HS), which extends the classical HS by considering the polarization state of light in the reciprocal paths. With the additional phase information from polarization, polar-HS requires only one reciprocal image pair. We derive the reciprocity relationship of Mueller matrix and formulate new reciprocity constraint that takes polarization state into account. We also utilize polarimetric constraints and extend them to the case of perspective projection. For the recovery of surface depths and normals, we incorporate reciprocity constraint with diffuse/specular polarimetric constraints in a unified optimization framework. For depth estimation, we further propose to utilize the consistency of diffuse angle of polarization. For normal estimation, we develop a normal refinement strategy based on degree of linear polarization. Using a hardware prototype, we show that our approach produces high-quality 3D reconstruction for different types of surfaces, ranging from diffuse to highly specular.
- 25 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (7) , 4597-4611
Polarimetric Helmholtz Stereopsis
Helmholtz stereopsis (HS) exploits the reciprocity principle of light propagation (i.e., the Helmholtz reciprocity) for 3D reconstruction of surfaces with arbitrary reflectance. In this paper, we present the polarimetric Helmholtz stereopsis (polar-HS), which extends the classical HS by considering the polarization state of light in the reciprocal paths. With the additional phase information from polarization, polar-HS requires only one reciprocal image pair. We derive the reciprocity relationship of Mueller matrix and formulate new reciprocity constraint that takes polarization state into account. We also utilize polarimetric constraints and extend them to the case of perspective projection. For the recovery of surface depths and normals, we incorporate reciprocity constraint with diffuse/specular polarimetric constraints in a unified optimization framework. For depth estimation, we further propose to utilize the consistency of diffuse angle of polarization. For normal estimation, we develop a normal refinement strategy based on degree of linear polarization. Using a hardware prototype, we show that our approach produces high-quality 3D reconstruction for different types of surfaces, ranging from diffuse to highly specular.
- 27 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 9 (25730436) , 881-893
Light Field-Based Underwater 3D Reconstruction via Angular Re-Sampling
Recovering 3D geometry of underwater scenes is challenging because of the non-linear refraction of light at the water-air interface, which is caused by the camera housing. We present a light field-based approach that leverages properties of angular samples for high-quality underwater 3D reconstruction from a single viewpoint. Specifically, we re-sample the light field image to angular patches. As underwater scenes exhibit weak view-dependent specularity, an angular patch tends to have uniform intensity when sampled at the correct depth. We thus impose this angular uniformity as a constraint for depth estimation. For efficient angular re-sampling, we design a fast approximation algorithm based on multivariate polynomial regression to approximate nonlinear refraction paths. We further develop a light field calibration algorithm that estimates the water-air interface geometry along with the camera parameters. Comprehensive experiments on synthetic and real data show our method produces state-of-the-art reconstruction of static and dynamic underwater scenes.
- 27 September 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 9 (25730436) , 881-893
Light Field-Based Underwater 3D Reconstruction via Angular Re-Sampling
Recovering 3D geometry of underwater scenes is challenging because of the non-linear refraction of light at the water-air interface, which is caused by the camera housing. We present a light field-based approach that leverages properties of angular samples for high-quality underwater 3D reconstruction from a single viewpoint. Specifically, we re-sample the light field image to angular patches. As underwater scenes exhibit weak view-dependent specularity, an angular patch tends to have uniform intensity when sampled at the correct depth. We thus impose this angular uniformity as a constraint for depth estimation. For efficient angular re-sampling, we design a fast approximation algorithm based on multivariate polynomial regression to approximate nonlinear refraction paths. We further develop a light field calibration algorithm that estimates the water-air interface geometry along with the camera parameters. Comprehensive experiments on synthetic and real data show our method produces state-of-the-art reconstruction of static and dynamic underwater scenes.
- Open Access
- 9 May 2023
- Published by Association for Computing Machinery (ACM)
- p. 370-371
Demo Abstract: Seamless High-Speed Optical Communication for Mobile Wide-Area Using Diffused Infrared Laser | Proceedings of the 22nd International Conference on Information Processing in Sensor Networks
We present a demo showcasing the capabilities of an infrared (IR)-based light communication system with a movable receiver. The system employs IR laser (VCSEL) together with scattering lens as the transmitter and an avalanche photo-diode (APD) with collimator as the receiver, using the reflection cross section existing in the environment (ceilings, walls, etc.) to spread the coverage of the communication system. The demonstration involves transmitting messages encoded as binary data through modulated IR light signals to the movable receiver. The receiver captures the signals using the APD, which are then decoded to retrieve the original message. The demonstration aims to showcase the robustness of the system against various sources of interference and the flexibility of the movable receiver to capture signals from different angles and positions. We believe that our demonstration will be useful for showcasing the potential of IR-based light communication in various applications (e.g., wireless VR/AR and high speed reliable data link) over more traditional communication methods that have limitations such as privacy and bandwidth.
- 9 May 2023
- Published by Association for Computing Machinery (ACM)
- p. 370-371
Open AccessDemo Abstract: Seamless High-Speed Optical Communication for Mobile Wide-Area Using Diffused Infrared Laser | Proceedings of the 22nd International Conference on Information Processing in Sensor Networks
We present a demo showcasing the capabilities of an infrared (IR)-based light communication system with a movable receiver. The system employs IR laser (VCSEL) together with scattering lens as the transmitter and an avalanche photo-diode (APD) with collimator as the receiver, using the reflection cross section existing in the environment (ceilings, walls, etc.) to spread the coverage of the communication system. The demonstration involves transmitting messages encoded as binary data through modulated IR light signals to the movable receiver. The receiver captures the signals using the APD, which are then decoded to retrieve the original message. The demonstration aims to showcase the robustness of the system against various sources of interference and the flexibility of the movable receiver to capture signals from different angles and positions. We believe that our demonstration will be useful for showcasing the potential of IR-based light communication in various applications (e.g., wireless VR/AR and high speed reliable data link) over more traditional communication methods that have limitations such as privacy and bandwidth.