Di Xu
Research Subjects
Di Xu has a total of 42 co-authors. Here are the top co-authors by number of shared publications:
- Haozhe Jia
Huawei Cloud Computing Technologies Company, Ltd., Guiyang, China
2 shared publications - Hengfei Cui
Northwestern Polytechnical University
2 shared publications - Qi Cao
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
2 shared publications - Tao Yu
BNRist, Tsinghua University, Beijing, China
2 shared publications Wenhan Zhu
Xueshen AI, Shanghai, China
2 shared publications- Yan Li
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science, Northwestern Polytechnical University, Xian, China
2 shared publications
Latest Publications
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (11) , 16859-16873
Adaptive Quantization for Stable Knowledge Acquisition in Quantization-Aware Continual Learning
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (11) , 16859-16873
Adaptive Quantization for Stable Knowledge Acquisition in Quantization-Aware Continual Learning
- 3 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2312-2328
Topo4D++: Realistic Physically Based 4D Head Capture With Topology-Preserving Gaussian Splatting and Expression Priors
4D head capture aims to generate dynamic facial meshes in the same topology with corresponding UV maps, which requires temporal correspondence between 3D head models. Existing pipelines either involve manual processing of artists or employ constraints such as landmark tracking and optical flow, failing to achieve a trade-off between accuracy and efficiency. To enhance this process, we propose Topo4D++, a novel framework for automatic geometry and texture reconstruction that optimizes densely aligned 4D heads and 8 K BRDF maps directly from calibrated multi-view videos. Our key insight is to represent facial models as a set of dynamic 3D Gaussians with fixed topology, where the Gaussian centers are bound to the mesh vertices. This enables tracking all vertices rather than sparse vertices on the face accurately by leveraging the inverse rendering capabilities of 3D Gaussian Splatting (3DGS), while also enabling ultra-high-resolution texture generation. To maintain face structure during dynamic 3DGS optimization, we propose to optimize geometry and texture alternatively under physical and topological constraints frame-by-frame and employ blendshape-based expression priors to address extreme expressions. Then, we propose to extract dynamic facial meshes in a regular wiring arrangement and high-fidelity textures with pore-level details from the learned Gaussians. Finally, we train a diffusion-based model to generate BRDF texture maps to achieve physically based rendering. Given the absence of a universal benchmark, we construct JHead, a novel benchmark for the comprehensive evaluation of 4D head capture methods. Extensive experiments on different datasets demonstrate that our method is generalized to different capture systems, identities, and expressions, outperforming current state-of-the-art head reconstruction methods in both mesh and texture qualitatively and quantitatively.
- 3 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2312-2328
Topo4D++: Realistic Physically Based 4D Head Capture With Topology-Preserving Gaussian Splatting and Expression Priors
4D head capture aims to generate dynamic facial meshes in the same topology with corresponding UV maps, which requires temporal correspondence between 3D head models. Existing pipelines either involve manual processing of artists or employ constraints such as landmark tracking and optical flow, failing to achieve a trade-off between accuracy and efficiency. To enhance this process, we propose Topo4D++, a novel framework for automatic geometry and texture reconstruction that optimizes densely aligned 4D heads and 8 K BRDF maps directly from calibrated multi-view videos. Our key insight is to represent facial models as a set of dynamic 3D Gaussians with fixed topology, where the Gaussian centers are bound to the mesh vertices. This enables tracking all vertices rather than sparse vertices on the face accurately by leveraging the inverse rendering capabilities of 3D Gaussian Splatting (3DGS), while also enabling ultra-high-resolution texture generation. To maintain face structure during dynamic 3DGS optimization, we propose to optimize geometry and texture alternatively under physical and topological constraints frame-by-frame and employ blendshape-based expression priors to address extreme expressions. Then, we propose to extract dynamic facial meshes in a regular wiring arrangement and high-fidelity textures with pore-level details from the learned Gaussians. Finally, we train a diffusion-based model to generate BRDF texture maps to achieve physically based rendering. Given the absence of a universal benchmark, we construct JHead, a novel benchmark for the comprehensive evaluation of 4D head capture methods. Extensive experiments on different datasets demonstrate that our method is generalized to different capture systems, identities, and expressions, outperforming current state-of-the-art head reconstruction methods in both mesh and texture qualitatively and quantitatively.
- 4 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (9) , 6053-6066
Super-NeRF: View-Consistent Detail Generation for NeRF Super-Resolution
The neural radiance field (NeRF) achieved remarkable success in modeling 3D scenes and synthesizing high-fidelity novel views. However, existing NeRF-based methods focus more on making full use of high-resolution images to generate high-resolution novel views, but less considering the generation of high-resolution details given only low-resolution images. In analogy to the extensive usage of image super-resolution, NeRF super-resolution is an effective way to generate low-resolution-guided high-resolution 3D scenes and holds great potential applications. Up to now, such an important topic is still under-explored. In this article, we propose a NeRF super-resolution method, named Super-NeRF, to generate high-resolution NeRF from only low-resolution inputs. Given multi-view low-resolution images, Super-NeRF constructs a multi-view consistency-controlling super-resolution module to generate various view-consistent high-resolution details for NeRF. Specifically, an optimizable latent code is introduced for each input view to control the generated reasonable high-resolution 2D images satisfying view consistency. The latent codes of each low-resolution image are optimized synergistically with the target Super-NeRF representation to utilize the view consistency constraint inherent in NeRF construction. We verify the effectiveness of Super-NeRF on synthetic, real-world, and even AI-generated NeRFs. Super-NeRF achieves state-of-the-art NeRF super-resolution performance on high-resolution detail generation and cross-view consistency.
- 4 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (9) , 6053-6066
Super-NeRF: View-Consistent Detail Generation for NeRF Super-Resolution
The neural radiance field (NeRF) achieved remarkable success in modeling 3D scenes and synthesizing high-fidelity novel views. However, existing NeRF-based methods focus more on making full use of high-resolution images to generate high-resolution novel views, but less considering the generation of high-resolution details given only low-resolution images. In analogy to the extensive usage of image super-resolution, NeRF super-resolution is an effective way to generate low-resolution-guided high-resolution 3D scenes and holds great potential applications. Up to now, such an important topic is still under-explored. In this article, we propose a NeRF super-resolution method, named Super-NeRF, to generate high-resolution NeRF from only low-resolution inputs. Given multi-view low-resolution images, Super-NeRF constructs a multi-view consistency-controlling super-resolution module to generate various view-consistent high-resolution details for NeRF. Specifically, an optimizable latent code is introduced for each input view to control the generated reasonable high-resolution 2D images satisfying view consistency. The latent codes of each low-resolution image are optimized synergistically with the target Super-NeRF representation to utilize the view consistency constraint inherent in NeRF construction. We verify the effectiveness of Super-NeRF on synthetic, real-world, and even AI-generated NeRFs. Super-NeRF achieves state-of-the-art NeRF super-resolution performance on high-resolution detail generation and cross-view consistency.
- 31 October 2024
- Published by Springer Nature
- p. 128-145
Topo4D: Topology-Preserving Gaussian Splatting for High-fidelity 4D Head Capture
- 31 October 2024
- Published by Springer Nature
- p. 128-145
Topo4D: Topology-Preserving Gaussian Splatting for High-fidelity 4D Head Capture
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 6793-6802
DisControlFace: Adding Disentangled Control to Diffusion Autoencoder for One-shot Explicit Facial Image Editing
In this work, we focus on exploring explicit fine-grained control of generative facial image editing, all while generating faithful facial appearances and consistent semantic details, which however, is quite challenging and has not been extensively explored, especially under an one-shot scenario. We identify the key challenge as the exploration of disentangled conditional control between high-level semantics and explicit parameters ( e.g., 3DMM) in the generation process, and accordingly propose a novel diffusion-based editing framework, named DisControlFace. Specifically, we leverage a Diffusion Autoencoder (Diff-AE) as the semantic reconstruction backbone. To enable explicit face editing, we construct an Exp-FaceNet that is compatible with Diff-AE to generate spatial-wise explicit control conditions based on estimated 3DMM parameters. Different from current diffusion-based editing methods that train the whole conditional generative model from scratch, we freeze the pre-trained weights of the Diff-AE to maintain its semantically deterministic conditioning capability and accordingly propose a random semantic masking (RSM) strategy to effectively achieve an independent training of Exp-FaceNet. This setting endows the model with disentangled face control meanwhile reducing semantic information shift in editing. Our model can be trained using 2D in-the-wild portrait images without requiring 3D or video data and perform robust editing on any new facial image through a simple one-shot fine-tuning. Comprehensive experiments demonstrate that DisControlFace can generate realistic facial images with better editing accuracy and identity preservation over SOTA methods.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 6793-6802
Open AccessDisControlFace: Adding Disentangled Control to Diffusion Autoencoder for One-shot Explicit Facial Image Editing
In this work, we focus on exploring explicit fine-grained control of generative facial image editing, all while generating faithful facial appearances and consistent semantic details, which however, is quite challenging and has not been extensively explored, especially under an one-shot scenario. We identify the key challenge as the exploration of disentangled conditional control between high-level semantics and explicit parameters ( e.g., 3DMM) in the generation process, and accordingly propose a novel diffusion-based editing framework, named DisControlFace. Specifically, we leverage a Diffusion Autoencoder (Diff-AE) as the semantic reconstruction backbone. To enable explicit face editing, we construct an Exp-FaceNet that is compatible with Diff-AE to generate spatial-wise explicit control conditions based on estimated 3DMM parameters. Different from current diffusion-based editing methods that train the whole conditional generative model from scratch, we freeze the pre-trained weights of the Diff-AE to maintain its semantically deterministic conditioning capability and accordingly propose a random semantic masking (RSM) strategy to effectively achieve an independent training of Exp-FaceNet. This setting endows the model with disentangled face control meanwhile reducing semantic information shift in editing. Our model can be trained using 2D in-the-wild portrait images without requiring 3D or video data and perform robust editing on any new facial image through a simple one-shot fine-tuning. Comprehensive experiments demonstrate that DisControlFace can generate realistic facial images with better editing accuracy and identity preservation over SOTA methods.
- 28 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 43 (8) , 2924-2936
Toward Accurate Cardiac MRI Segmentation With Variational Autoencoder-Based Unsupervised Domain Adaptation
Accurate myocardial segmentation is crucial in the diagnosis and treatment of myocardial infarction (MI), especially in Late Gadolinium Enhancement (LGE) cardiac magnetic resonance (CMR) images, where the infarcted myocardium exhibits a greater brightness. However, segmentation annotations for LGE images are usually not available. Although knowledge gained from CMR images of other modalities with ample annotations, such as balanced-Steady State Free Precession (bSSFP), can be transferred to the LGE images, the difference in image distribution between the two modalities (i.e., domain shift) usually results in a significant degradation in model performance. To alleviate this, an end-to-end Variational autoencoder based feature Alignment Module Combining Explicit and Implicit features (VAMCEI) is proposed. We first re-derive the Kullback-Leibler (KL) divergence between the posterior distributions of the two domains as a measure of the global distribution distance. Second, we calculate the prototype contrastive loss between the two domains, bringing closer the prototypes of the same category across domains and pushing away the prototypes of different categories within or across domains. Finally, a domain discriminator is added to the output space, which indirectly aligns the feature distribution and forces the extracted features to be more favorable for segmentation. In addition, by combining CycleGAN and VAMCEI, we propose a more refined multi-stage unsupervised domain adaptation (UDA) framework for myocardial structure segmentation. We conduct extensive experiments on the MSCMRSeg 2019, MyoPS 2020 and MM-WHS 2017 datasets. The experimental results demonstrate that our framework achieves superior performances than state-of-the-art methods.
- 28 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 43 (8) , 2924-2936
Toward Accurate Cardiac MRI Segmentation With Variational Autoencoder-Based Unsupervised Domain Adaptation
Accurate myocardial segmentation is crucial in the diagnosis and treatment of myocardial infarction (MI), especially in Late Gadolinium Enhancement (LGE) cardiac magnetic resonance (CMR) images, where the infarcted myocardium exhibits a greater brightness. However, segmentation annotations for LGE images are usually not available. Although knowledge gained from CMR images of other modalities with ample annotations, such as balanced-Steady State Free Precession (bSSFP), can be transferred to the LGE images, the difference in image distribution between the two modalities (i.e., domain shift) usually results in a significant degradation in model performance. To alleviate this, an end-to-end Variational autoencoder based feature Alignment Module Combining Explicit and Implicit features (VAMCEI) is proposed. We first re-derive the Kullback-Leibler (KL) divergence between the posterior distributions of the two domains as a measure of the global distribution distance. Second, we calculate the prototype contrastive loss between the two domains, bringing closer the prototypes of the same category across domains and pushing away the prototypes of different categories within or across domains. Finally, a domain discriminator is added to the output space, which indirectly aligns the feature distribution and forces the extracted features to be more favorable for segmentation. In addition, by combining CycleGAN and VAMCEI, we propose a more refined multi-stage unsupervised domain adaptation (UDA) framework for myocardial structure segmentation. We conduct extensive experiments on the MSCMRSeg 2019, MyoPS 2020 and MM-WHS 2017 datasets. The experimental results demonstrate that our framework achieves superior performances than state-of-the-art methods.
- Open Access
- 17 August 2022
- Published by MDPI AG in Electronics
- Vol. 11 (16) , 2568
Serious Game for VR Road Crossing in Special Needs Education
Autism spectrum disorder (ASD) is a developmental disability that can impair communication, social skills, living skills, and learning capabilities. Learning approaches usually differ between mainstream schools and special needs schools, to cater for the different learning processes of children with ASD. Besides the traditional classroom-based education, alternative technology and methods are explored for special needs education. One method is to train children with ASD using Virtual Reality (VR) technologies. Many prior works show the effectiveness of VR-based learning with varying degrees of success. Some children with ASD may face challenges to gain independent living skills. Their parents or guardians have to expend a significant amount of effort in taking care of children with ASD. It will be very helpful if they can have a learning opportunity to gain such living skills. In this research, we develop a VR serious game to train children with ASD one of the basic living skills for road crossing safely. The VR serious game operates on multiple types of platforms, with various user interaction inputs including the Microsoft Kinect sensor, keyboard, mouse, and touch screen. The game design and methodology will be described in this paper. Experiments have been conducted to evaluate the learning effectiveness of the road crossing game, with very positive results achieved in the quiz and survey questionnaire after the gameplay.
- 17 August 2022
- Published by MDPI AG in Electronics
- Vol. 11 (16) , 2568
Open AccessSerious Game for VR Road Crossing in Special Needs Education
Autism spectrum disorder (ASD) is a developmental disability that can impair communication, social skills, living skills, and learning capabilities. Learning approaches usually differ between mainstream schools and special needs schools, to cater for the different learning processes of children with ASD. Besides the traditional classroom-based education, alternative technology and methods are explored for special needs education. One method is to train children with ASD using Virtual Reality (VR) technologies. Many prior works show the effectiveness of VR-based learning with varying degrees of success. Some children with ASD may face challenges to gain independent living skills. Their parents or guardians have to expend a significant amount of effort in taking care of children with ASD. It will be very helpful if they can have a learning opportunity to gain such living skills. In this research, we develop a VR serious game to train children with ASD one of the basic living skills for road crossing safely. The VR serious game operates on multiple types of platforms, with various user interaction inputs including the Microsoft Kinect sensor, keyboard, mouse, and touch screen. The game design and methodology will be described in this paper. Experiments have been conducted to evaluate the learning effectiveness of the road crossing game, with very positive results achieved in the quiz and survey questionnaire after the gameplay.
- 5 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 383-389
Interactive Virtual Reality Game for Online Learning of Science Subject in Primary Schools
Education plays an important role in nurturing children. COVID-19 pandemic brings challenges or disruptions to school education, due to school closures in some countries. Science subject in primary schools is unique as hands-on experiments are important learning components. Its learning process may be affected, as a new norm of online learning or home-based learning. This research project creates a serious game on science subject for primary school students aging within 10 to 11 years old using virtual reality (VR) technology. It consists of three virtual learning phases. Phase 1 explains theories of science topics on electricity and electric circuits. Phase 2 provides interactive hands-on experiment exercises where students can practice theory knowledge learned in the previous phase. An interactive quiz session is offered to reinforce the learning in Phase 3. Interactive VR features enable primary school students learning abstract science concepts in an interesting way compared to conventional classroom settings. Meticulous design attentions have been placed in the details such as visual instructions, voice instructions, speech tempo, animations, and colorful graphics to create a sense of realism and keep students actively engaged. Preliminary case study has been conducted with 10 students at primary schools in Singapore to evaluate learning effectiveness in this research.
- 5 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 383-389
Interactive Virtual Reality Game for Online Learning of Science Subject in Primary Schools
Education plays an important role in nurturing children. COVID-19 pandemic brings challenges or disruptions to school education, due to school closures in some countries. Science subject in primary schools is unique as hands-on experiments are important learning components. Its learning process may be affected, as a new norm of online learning or home-based learning. This research project creates a serious game on science subject for primary school students aging within 10 to 11 years old using virtual reality (VR) technology. It consists of three virtual learning phases. Phase 1 explains theories of science topics on electricity and electric circuits. Phase 2 provides interactive hands-on experiment exercises where students can practice theory knowledge learned in the previous phase. An interactive quiz session is offered to reinforce the learning in Phase 3. Interactive VR features enable primary school students learning abstract science concepts in an interesting way compared to conventional classroom settings. Meticulous design attentions have been placed in the details such as visual instructions, voice instructions, speech tempo, animations, and colorful graphics to create a sense of realism and keep students actively engaged. Preliminary case study has been conducted with 10 students at primary schools in Singapore to evaluate learning effectiveness in this research.
- 17 February 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 40 (2) , 423-436
Shading-Based Surface Detail Recovery Under General Unknown Illumination
Reconstructing the shape of a 3D object from multi-view images under unknown, general illumination is a fundamental problem in computer vision. High quality reconstruction is usually challenging especially when fine detail is needed and the albedo of the object is non-uniform. This paper introduces vertex overall illumination vectors to model the illumination effect and presents a total variation (TV) based approach for recovering surface details using shading and multi-view stereo (MVS). Behind the approach are the two important observations: (1) the illumination over the surface of an object often appears to be piecewise smooth and (2) the recovery of surface orientation is not sufficient for reconstructing the surface, which was often overlooked previously. Thus we propose to use TV to regularize the overall illumination vectors and use visual hull to constrain partial vertices. The reconstruction is formulated as a constrained TV-minimization problem that simultaneously treats the shape and illumination vectors as unknowns. An augmented Lagrangian method is proposed to quickly solve the TV-minimization problem. As a result, our approach is robust, stable and is able to efficiently recover high-quality surface details even when starting with a coarse model obtained using MVS. These advantages are demonstrated by extensive experiments on the state-of-the-art MVS database, which includes challenging objects with varying albedo.
- 17 February 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 40 (2) , 423-436
Shading-Based Surface Detail Recovery Under General Unknown Illumination
Reconstructing the shape of a 3D object from multi-view images under unknown, general illumination is a fundamental problem in computer vision. High quality reconstruction is usually challenging especially when fine detail is needed and the albedo of the object is non-uniform. This paper introduces vertex overall illumination vectors to model the illumination effect and presents a total variation (TV) based approach for recovering surface details using shading and multi-view stereo (MVS). Behind the approach are the two important observations: (1) the illumination over the surface of an object often appears to be piecewise smooth and (2) the recovery of surface orientation is not sufficient for reconstructing the surface, which was often overlooked previously. Thus we propose to use TV to regularize the overall illumination vectors and use visual hull to constrain partial vertices. The reconstruction is formulated as a constrained TV-minimization problem that simultaneously treats the shape and illumination vectors as unknowns. An augmented Lagrangian method is proposed to quickly solve the TV-minimization problem. As a result, our approach is robust, stable and is able to efficiently recover high-quality surface details even when starting with a coarse model obtained using MVS. These advantages are demonstrated by extensive experiments on the state-of-the-art MVS database, which includes challenging objects with varying albedo.
- Open Access
- 1 August 2006
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21627843,p. 868-873
Design of High-Performance Filter Banks for Image Coding
An optimization-based method for the design of high-performance separable filter banks for image coding is proposed. This method yields linear-phase perfect-reconstruction systems with high coding gain, good frequency selectivity, and certain prescribed vanishing-moment properties. Several filter banks designed with the proposed method are presented and shown to work extremely well for image coding, outperforming the well-known 9/7 filter bank from the JPEG-2000 standard in most cases
- 1 August 2006
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21627843,p. 868-873
Open AccessDesign of High-Performance Filter Banks for Image Coding
An optimization-based method for the design of high-performance separable filter banks for image coding is proposed. This method yields linear-phase perfect-reconstruction systems with high coding gain, good frequency selectivity, and certain prescribed vanishing-moment properties. Several filter banks designed with the proposed method are presented and shown to work extremely well for image coding, outperforming the well-known 9/7 filter bank from the JPEG-2000 standard in most cases