Tat-Jen Cham
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Tat-Jen Cham has a total of 161 co-authors. Here are the top co-authors by number of shared publications:
- Jianfei Cai
Monash University, Melbourne, VIC, Australia
39 shared publications - Chuanxia Zheng
Visual Geometry Group University of Oxford
15 shared publications - Jianmin Zheng
Nanyang technological university, Singapore, Singapore
7 shared publications Jiwen Lu
Tsinghua University, China
7 shared publications- Anran Wang
Nanyang Technological University
6 shared publications - Roberto Cipolla
University of Cambridge
6 shared publications
Latest Publications
- Open Access
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Pinlight AR Near-Eye Display with Large FOV and Expanded Eyebox through Eye Tracking
We have developed a novel design of a compact pinlight near-eye display for augmented reality (AR) featuring a wide field of view (FOV) and an expanded eyebox through dynamic eye tracking. In our design, a camera rig mounted on the pinlight AR NED captures multi-view observations of the viewer’s pupil, and the captured pupil images are utilized to predict the 3D pupil location through a trained neural network. The 3D pupil location is then used to calculate a display pattern that spatio-temporally modulates the light entering the viewer’s eyes. A prototype following the design is developed, with preliminary testing conducted, showing the effectiveness of our proposed method.
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessPinlight AR Near-Eye Display with Large FOV and Expanded Eyebox through Eye Tracking
We have developed a novel design of a compact pinlight near-eye display for augmented reality (AR) featuring a wide field of view (FOV) and an expanded eyebox through dynamic eye tracking. In our design, a camera rig mounted on the pinlight AR NED captures multi-view observations of the viewer’s pupil, and the captured pupil images are utilized to predict the 3D pupil location through a trained neural network. The 3D pupil location is then used to calculate a display pattern that spatio-temporally modulates the light entering the viewer’s eyes. A prototype following the design is developed, with preliminary testing conducted, showing the effectiveness of our proposed method.
- 3 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (11) , 10016-10025
Dynamic Eyebox Steering for Improved Pinlight AR Near-Eye Displays
An optical-see-through near-eye display (NED) for augmented reality (AR) allows the user to perceive virtual and real imagery simultaneously. Existing technologies for optical-see-through AR NEDs involve trade-offs between key metrics such as field of view (FOV), eyebox size, form factor, etc. We have enhanced an existing compact wide-FOV pinlight AR NED design with real-time 3D pupil localization in order to dynamically steer and thus effectively enlarge the usable eyebox. This is achieved with a dual-camera rig that captures stereoscopic views of the pupils. The 3D pupil location is used to dynamically calculate a display pattern that spatio-temporally modulates the light entering the wearer's eyes. We have built a demonstrable compact prototype and have conducted a user study that indicates the effectiveness of our eyebox steering method (e.g., without eyebox steering, in 10.5% of our tests, users were unable to perceive the test pattern correctly before experiment timeout; with eyebox steering, that fraction decreased dramatically to 1.25%). This is a small yet crucial step in making simple wide-FOV pinlight NEDs usable for human users and not just as demonstration prototypes filmed with a precisely positioned camera standing in for the user's eye. Further contributions of this paper include a detailed description of display design, calibration technique, and user study design, all of which may benefit other NED research.
- 3 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (11) , 10016-10025
Dynamic Eyebox Steering for Improved Pinlight AR Near-Eye Displays
An optical-see-through near-eye display (NED) for augmented reality (AR) allows the user to perceive virtual and real imagery simultaneously. Existing technologies for optical-see-through AR NEDs involve trade-offs between key metrics such as field of view (FOV), eyebox size, form factor, etc. We have enhanced an existing compact wide-FOV pinlight AR NED design with real-time 3D pupil localization in order to dynamically steer and thus effectively enlarge the usable eyebox. This is achieved with a dual-camera rig that captures stereoscopic views of the pupils. The 3D pupil location is used to dynamically calculate a display pattern that spatio-temporally modulates the light entering the wearer's eyes. We have built a demonstrable compact prototype and have conducted a user study that indicates the effectiveness of our eyebox steering method (e.g., without eyebox steering, in 10.5% of our tests, users were unable to perceive the test pattern correctly before experiment timeout; with eyebox steering, that fraction decreased dramatically to 1.25%). This is a small yet crucial step in making simple wide-FOV pinlight NEDs usable for human users and not just as demonstration prototypes filmed with a precisely positioned camera standing in for the user's eye. Further contributions of this paper include a detailed description of display design, calibration technique, and user study design, all of which may benefit other NED research.
- Top Cited Papers
- 13 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (12) , 11708-11719
Explicit Correspondence Matching for Generalizable Neural Radiance Fields
We present a new generalizable NeRF method that is able to directly generalize to new unseen scenarios and perform novel view synthesis with as few as two source views. The key to our approach lies in the explicitly modeled correspondence matching information, so as to provide the geometry prior to the prediction of NeRF color and density for volume rendering. The explicit correspondence matching is quantified with the cosine similarity between image features sampled at the 2D projections of a 3D point on different views, which is able to provide reliable cues about the surface geometry. Unlike previous methods where image features are extracted independently for each view, we consider modeling the cross-view interactions via Transformer cross-attention, which greatly improves the feature matching quality. Our method achieves state-of-the-art results on different evaluation settings, with the experiments showing a strong correlation between our learned cosine feature similarity and volume density, demonstrating the effectiveness and superiority of our proposed method.
- 13 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (12) , 11708-11719
Top Cited PapersExplicit Correspondence Matching for Generalizable Neural Radiance Fields
We present a new generalizable NeRF method that is able to directly generalize to new unseen scenarios and perform novel view synthesis with as few as two source views. The key to our approach lies in the explicitly modeled correspondence matching information, so as to provide the geometry prior to the prediction of NeRF color and density for volume rendering. The explicit correspondence matching is quantified with the cosine similarity between image features sampled at the 2D projections of a 3D point on different views, which is able to provide reliable cues about the surface geometry. Unlike previous methods where image features are extracted independently for each view, we consider modeling the cross-view interactions via Transformer cross-attention, which greatly improves the feature matching quality. Our method achieves state-of-the-art results on different evaluation settings, with the experiments showing a strong correlation between our learned cosine feature similarity and volume density, demonstrating the effectiveness and superiority of our proposed method.
- Open Access
- 28 July 2025
- Published by Radcliffe Medical Media Ltd in Journal of Asian Pacific Society of Cardiology
- Vol. 4
Computational Methods for Analysing X-ray-guided Coronary Angiography: From Key Frame Selection to Vessel Segmentation, Stenosis Detection and Classification
Coronary artery disease (CAD) is a leading cause of mortality globally, highlighting the need for accurate diagnostic tools. X-ray-guided coronary angiography is the foundation for assessing CAD, involving three key tasks: key frame selection, vessel segmentation and stenosis detection. This review summarises conventional and advanced artificial intelligence (AI) methods applied to these tasks. AI’s integration is vital for enhancing efficiency and reproducibility, allowing real-time analysis and reducing manual labour. However, challenges remain, including the need for large, annotated datasets and difficulties in implementing AI systems in clinical trials. We discuss these challenges and the limitations of current AI approaches, exploring future directions for AI-driven solutions. Ultimately, we provide insights into AI’s potential to improve the precision and efficiency of coronary angiography analysis and CAD diagnosis, contributing to better clinical outcomes and advancing cardiovascular medicine.
- 28 July 2025
- Published by Radcliffe Medical Media Ltd in Journal of Asian Pacific Society of Cardiology
- Vol. 4
Open AccessComputational Methods for Analysing X-ray-guided Coronary Angiography: From Key Frame Selection to Vessel Segmentation, Stenosis Detection and Classification
Coronary artery disease (CAD) is a leading cause of mortality globally, highlighting the need for accurate diagnostic tools. X-ray-guided coronary angiography is the foundation for assessing CAD, involving three key tasks: key frame selection, vessel segmentation and stenosis detection. This review summarises conventional and advanced artificial intelligence (AI) methods applied to these tasks. AI’s integration is vital for enhancing efficiency and reproducibility, allowing real-time analysis and reducing manual labour. However, challenges remain, including the need for large, annotated datasets and difficulties in implementing AI systems in clinical trials. We discuss these challenges and the limitations of current AI approaches, exploring future directions for AI-driven solutions. Ultimately, we provide insights into AI’s potential to improve the precision and efficiency of coronary angiography analysis and CAD diagnosis, contributing to better clinical outcomes and advancing cardiovascular medicine.
- Open Access
Semantix: An Energy Guided Sampler for Semantic Style Transfer
Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, Semantic Style Transfer, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, Semantix an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, Semantix be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, Semantix utilizes a meticulously crafted energy function to guide the sampling process, including three key components: Style Feature Guidance, Spatial Feature Guidance and Semantic Distance as a regularisation term. Experimental results demonstrate that Semantix not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields. The project website is available at https://huiang-he.github.io/semantix/
Open AccessSemantix: An Energy Guided Sampler for Semantic Style Transfer
Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, Semantic Style Transfer, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, Semantix an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, Semantix be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, Semantix utilizes a meticulously crafted energy function to guide the sampling process, including three key components: Style Feature Guidance, Spatial Feature Guidance and Semantic Distance as a regularisation term. Experimental results demonstrate that Semantix not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields. The project website is available at https://huiang-he.github.io/semantix/
- Open Access
Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images
Most image-based 3D object reconstructors assume that objects are fully visible, ignoring occlusions that commonly occur in real-world scenarios. In this paper, we introduce Amodal3R, a conditional 3D generative model designed to reconstruct 3D objects from partial observations. We start from a "foundation" 3D generative model and extend it to recover plausible 3D geometry and appearance from occluded objects. We introduce a mask-weighted multi-head cross-attention mechanism followed by an occlusion-aware attention layer that explicitly leverages occlusion priors to guide the reconstruction process. We demonstrate that, by training solely on synthetic data, Amodal3R learns to recover full 3D objects even in the presence of occlusions in real scenes. It substantially outperforms existing methods that independently perform 2D amodal completion followed by 3D reconstruction, thereby establishing a new benchmark for occlusion-aware 3D reconstruction.
Open AccessAmodal3R: Amodal 3D Reconstruction from Occluded 2D Images
Most image-based 3D object reconstructors assume that objects are fully visible, ignoring occlusions that commonly occur in real-world scenarios. In this paper, we introduce Amodal3R, a conditional 3D generative model designed to reconstruct 3D objects from partial observations. We start from a "foundation" 3D generative model and extend it to recover plausible 3D geometry and appearance from occluded objects. We introduce a mask-weighted multi-head cross-attention mechanism followed by an occlusion-aware attention layer that explicitly leverages occlusion priors to guide the reconstruction process. We demonstrate that, by training solely on synthetic data, Amodal3R learns to recover full 3D objects even in the presence of occlusions in real scenes. It substantially outperforms existing methods that independently perform 2D amodal completion followed by 3D reconstruction, thereby establishing a new benchmark for occlusion-aware 3D reconstruction.
- 27 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 3939-3952
One-Shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing
Human motion transfer aims at animating a static source image with a driving video. While recent advances in one-shot human motion transfer have led to significant improvement in results, it remains challenging for methods with 2D body landmarks, skeleton and semantic mask to accurately capture correspondences between source and driving poses due to the large variation in motion and articulation complexity. In addition, the accuracy and precision of DensePose degrade the image quality for neural-rendering-based methods. To address the limitations and by both considering the importance of appearance and geometry for motion transfer, in this work, we proposed a unified framework that combines multi-scale feature warping and neural texture mapping to recover better 2D appearance and 2.5D geometry, partly by exploiting the information from DensePose, yet adapting to its inherent limited accuracy. Our model takes advantage of multiple modalities by jointly training and fusing them, which allows it to robust neural texture features that cope with geometric errors as well as multi-scale dense motion flow that better preserves appearance. Experimental results with full and half-view body video datasets demonstrate that our model can generalize well and achieve competitive results, and that it is particularly effective in handling challenging cases such as those with substantial self-occlusions.
- 27 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 3939-3952
One-Shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing
Human motion transfer aims at animating a static source image with a driving video. While recent advances in one-shot human motion transfer have led to significant improvement in results, it remains challenging for methods with 2D body landmarks, skeleton and semantic mask to accurately capture correspondences between source and driving poses due to the large variation in motion and articulation complexity. In addition, the accuracy and precision of DensePose degrade the image quality for neural-rendering-based methods. To address the limitations and by both considering the importance of appearance and geometry for motion transfer, in this work, we proposed a unified framework that combines multi-scale feature warping and neural texture mapping to recover better 2D appearance and 2.5D geometry, partly by exploiting the information from DensePose, yet adapting to its inherent limited accuracy. Our model takes advantage of multiple modalities by jointly training and fusing them, which allows it to robust neural texture features that cope with geometric errors as well as multi-scale dense motion flow that better preserves appearance. Experimental results with full and half-view body video datasets demonstrate that our model can generalize well and achieve competitive results, and that it is particularly effective in handling challenging cases such as those with substantial self-occlusions.
- 5 December 2024
- Published by Springer Nature
- p. 143-159
3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting
- 5 December 2024
- Published by Springer Nature
- p. 143-159
3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting
- 3 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 55 (2) , 1374-1387
D-LORD for Motion Stylization
This article introduces a novel framework named double-latent optimization for representation disentanglement (D-LORD), which is designed for motion stylization (motion style transfer and motion retargeting). The primary objective of this framework is to separate the class and content information from a given motion sequence using a data-driven latent optimization approach. Here, class refers to person-specific style, such as a particular emotion or an individual’s identity, while content relates to the style-agnostic aspect of an action, such as walking or jumping, as universally understood concepts. The key advantage of D-LORD is its ability to perform style transfer without needing paired motion data. Instead, it utilizes class and content labels during the latent optimization process. By disentangling the representation, the framework enables the transformation of one motion sequence’s style to another’s style using adaptive instance normalization. The proposed D-LORD framework is designed with a focus on generalization, allowing it to handle different class and content labels for various applications. In addition, it can generate diverse motion sequences when specific class and content labels are provided. The framework’s efficacy is demonstrated through experimentation on three datasets: 1) the CMU XIA dataset for motion style transfer; 2) the multimodal human action database dataset; and 3) the RRIS Ability dataset for motion retargeting. Notably, this article presents the first generalized framework for motion style transfer and motion retargeting, showcasing its potential contributions in this area.
- 3 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 55 (2) , 1374-1387
D-LORD for Motion Stylization
This article introduces a novel framework named double-latent optimization for representation disentanglement (D-LORD), which is designed for motion stylization (motion style transfer and motion retargeting). The primary objective of this framework is to separate the class and content information from a given motion sequence using a data-driven latent optimization approach. Here, class refers to person-specific style, such as a particular emotion or an individual’s identity, while content relates to the style-agnostic aspect of an action, such as walking or jumping, as universally understood concepts. The key advantage of D-LORD is its ability to perform style transfer without needing paired motion data. Instead, it utilizes class and content labels during the latent optimization process. By disentangling the representation, the framework enables the transformation of one motion sequence’s style to another’s style using adaptive instance normalization. The proposed D-LORD framework is designed with a focus on generalization, allowing it to handle different class and content labels for various applications. In addition, it can generate diverse motion sequences when specific class and content labels are provided. The framework’s efficacy is demonstrated through experimentation on three datasets: 1) the CMU XIA dataset for motion style transfer; 2) the multimodal human action database dataset; and 3) the RRIS Ability dataset for motion retargeting. Notably, this article presents the first generalized framework for motion style transfer and motion retargeting, showcasing its potential contributions in this area.
- Top Cited Papers
- 26 October 2024
- Published by Springer Nature
- p. 370-386
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images
- 26 October 2024
- Published by Springer Nature
- p. 370-386
Top Cited PapersMVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images