Shihong Xia
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Shihong Xia has a total of 65 co-authors. Here are the top co-authors by number of shared publications:
- Lin Gao
University of Chinese Academy of Sciences, Beijing, China
8 shared publications - Zihao Zhang
Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, Chinese Academy of Sciences, Nanjing, China
8 shared publications - Lei Hu
Institute of Computing Technology Chinese Academy of Sciences
7 shared publications - Yu-Kun Lai
Cardiff University, United Kingdom
5 shared publications - Chongyang Zhong
Institute of Computing Technology Chinese Academy of Sciences
3 shared publications - Chunpeng Li
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
3 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-16
Capturing Rich Details of Human Eyes
Capturing rich details of human eyes is essential for realistic digital humans, yet remains challenging due to the eye's complex optical structure. The sclera, cornea, and iris exhibit distinct appearance properties, and their geometry and appearance are tightly coupled through reflection and refraction in image observations. We present a human eye capture and reconstruction system based on multi-source optical measurements, which are tailored to different eye components to capture detail-rich observations at multiple scales and provide direct constraints for staged, decoupled reconstruction. For the sclera, multi-view and multi-gaze imaging recovers subject-specific exterior geometry and establishes an eye-centric frame. For the cornea, specular highlights and dense Placido-based topographic cues estimate the anterior surface before it is used as the refractive interface for iris reconstruction. For the iris, refraction-aware photometric reconstruction recovers albedo, normals, and fine-scale displacement under limited viewing angles. Experiments demonstrate detailed reconstruction of scleral shape, corneal topography, and iris micro-geometry across diverse subjects. The reconstructed assets further expose physiology-supported ocular details, including scleral asymmetry, non-axisymmetric corneal shape, iris relief, and pupil-center motion, providing evidence for reconstruction fidelity and useful annotations for downstream rendering, tracking, and data generation. We will release a consented 20-eye reconstructed ocular asset dataset containing component models and textures for all eye components.
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-16
Open AccessCapturing Rich Details of Human Eyes
Capturing rich details of human eyes is essential for realistic digital humans, yet remains challenging due to the eye's complex optical structure. The sclera, cornea, and iris exhibit distinct appearance properties, and their geometry and appearance are tightly coupled through reflection and refraction in image observations. We present a human eye capture and reconstruction system based on multi-source optical measurements, which are tailored to different eye components to capture detail-rich observations at multiple scales and provide direct constraints for staged, decoupled reconstruction. For the sclera, multi-view and multi-gaze imaging recovers subject-specific exterior geometry and establishes an eye-centric frame. For the cornea, specular highlights and dense Placido-based topographic cues estimate the anterior surface before it is used as the refractive interface for iris reconstruction. For the iris, refraction-aware photometric reconstruction recovers albedo, normals, and fine-scale displacement under limited viewing angles. Experiments demonstrate detailed reconstruction of scleral shape, corneal topography, and iris micro-geometry across diverse subjects. The reconstructed assets further expose physiology-supported ocular details, including scleral asymmetry, non-axisymmetric corneal shape, iris relief, and pupil-center motion, providing evidence for reconstruction fidelity and useful annotations for downstream rendering, tracking, and data generation. We will release a consented 20-eye reconstructed ocular asset dataset containing component models and textures for all eye components.
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8432-8440
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8432-8440
Open AccessEstimating Accurate Hand Pose in Camera Space with Vision Transformer
- 1 June 2026
- Published by Springer Nature
- p. 243-259
Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Face Animation
- 1 June 2026
- Published by Springer Nature
- p. 243-259
Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Face Animation
- Open Access
Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation
Expressions are fundamental to conveying human emotions. With the rapid advancement of AI-generated content (AIGC), realistic and expressive 3D facial animation has become increasingly crucial. Despite recent progress in speech-driven lip-sync for talking-face animation, generating emotionally expressive talking faces remains underexplored. A major obstacle is the scarcity of real emotional 3D talking-face datasets due to the high cost of data capture. To address this, we model facial animation driven by both speech and emotion as a linear additive problem. Leveraging a 3D talking-face dataset with neutral expressions (VOCAset) and a dataset of 3D expression sequences (Florence4D), we jointly learn a set of blendshapes driven by speech and emotion. We introduce a sparsity constraint loss to encourage disentanglement between the two types of blendshapes while allowing the model to capture inherent secondary cross-domain deformations present in the training data. The learned blendshapes can be further mapped to the expression and jaw pose parameters of the FLAME model, enabling the animation of 3D Gaussian avatars. Qualitative and quantitative experiments demonstrate that our method naturally generates talking faces with specified expressions while maintaining accurate lip synchronization. Perceptual studies further show that our approach achieves superior emotional expressivity compared to existing methods, without compromising lip-sync quality.
Open AccessLearning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation
Expressions are fundamental to conveying human emotions. With the rapid advancement of AI-generated content (AIGC), realistic and expressive 3D facial animation has become increasingly crucial. Despite recent progress in speech-driven lip-sync for talking-face animation, generating emotionally expressive talking faces remains underexplored. A major obstacle is the scarcity of real emotional 3D talking-face datasets due to the high cost of data capture. To address this, we model facial animation driven by both speech and emotion as a linear additive problem. Leveraging a 3D talking-face dataset with neutral expressions (VOCAset) and a dataset of 3D expression sequences (Florence4D), we jointly learn a set of blendshapes driven by speech and emotion. We introduce a sparsity constraint loss to encourage disentanglement between the two types of blendshapes while allowing the model to capture inherent secondary cross-domain deformations present in the training data. The learned blendshapes can be further mapped to the expression and jaw pose parameters of the FLAME model, enabling the animation of 3D Gaussian avatars. Qualitative and quantitative experiments demonstrate that our method naturally generates talking faces with specified expressions while maintaining accurate lip synchronization. Perceptual studies further show that our approach achieves superior emotional expressivity compared to existing methods, without compromising lip-sync quality.
- 9 October 2024
- Published by Wiley in Computer Graphics Forum
- Vol. 43 (8)
Diffusion‐based Human Motion Style Transfer with Semantic Guidance
3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN‐based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN‐based methods. Therefore, in this paper, we propose a novel two‐stage framework for few‐shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre‐train a diffusion‐based text‐to‐motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine‐tune the pre‐trained diffusion model in a few‐shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion‐style translation process since the motion styles can be viewed as special motion variations. During the fine‐tuning for style transfer, a simple yet effective semantic‐guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state‐of‐the‐art performance and has practical applications. The source code is available at https://github.com/hlcdyy/diffusion-based-motion-style-transfer.
- 9 October 2024
- Published by Wiley in Computer Graphics Forum
- Vol. 43 (8)
Diffusion‐based Human Motion Style Transfer with Semantic Guidance
3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN‐based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN‐based methods. Therefore, in this paper, we propose a novel two‐stage framework for few‐shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre‐train a diffusion‐based text‐to‐motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine‐tune the pre‐trained diffusion model in a few‐shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion‐style translation process since the motion styles can be viewed as special motion variations. During the fine‐tuning for style transfer, a simple yet effective semantic‐guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state‐of‐the‐art performance and has practical applications. The source code is available at https://github.com/hlcdyy/diffusion-based-motion-style-transfer.
- 26 August 2023
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 83 (9) , 27901-27920
Mining collaborative spatio-temporal clues for face forgery detection
- 26 August 2023
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 83 (9) , 27901-27920
Mining collaborative spatio-temporal clues for face forgery detection
- Open Access
Pose-aware Attention Network for Flexible Motion Retargeting by Body Part
Motion retargeting is a fundamental problem in computer graphics and computer vision. Existing approaches usually have many strict requirements, such as the source-target skeletons needing to have the same number of joints or share the same topology. To tackle this problem, we note that skeletons with different structure may have some common body parts despite the differences in joint numbers. Following this observation, we propose a novel, flexible motion retargeting framework. The key idea of our method is to regard the body part as the basic retargeting unit rather than directly retargeting the whole body motion. To enhance the spatial modeling capability of the motion encoder, we introduce a pose-aware attention network (PAN) in the motion encoding phase. The PAN is pose-aware since it can dynamically predict the joint weights within each body part based on the input pose, and then construct a shared latent space for each body part by feature pooling. Extensive experiments show that our approach can generate better motion retargeting results both qualitatively and quantitatively than state-of-the-art methods. Moreover, we also show that our framework can generate reasonable results even for a more challenging retargeting scenario, like retargeting between bipedal and quadrupedal skeletons because of the body part retargeting strategy and PAN. Our code is publicly available.
Open AccessPose-aware Attention Network for Flexible Motion Retargeting by Body Part
Motion retargeting is a fundamental problem in computer graphics and computer vision. Existing approaches usually have many strict requirements, such as the source-target skeletons needing to have the same number of joints or share the same topology. To tackle this problem, we note that skeletons with different structure may have some common body parts despite the differences in joint numbers. Following this observation, we propose a novel, flexible motion retargeting framework. The key idea of our method is to regard the body part as the basic retargeting unit rather than directly retargeting the whole body motion. To enhance the spatial modeling capability of the motion encoder, we introduce a pose-aware attention network (PAN) in the motion encoding phase. The PAN is pose-aware since it can dynamically predict the joint weights within each body part based on the input pose, and then construct a shared latent space for each body part by feature pooling. Extensive experiments show that our approach can generate better motion retargeting results both qualitatively and quantitatively than state-of-the-art methods. Moreover, we also show that our framework can generate reasonable results even for a more challenging retargeting scenario, like retargeting between bipedal and quadrupedal skeletons because of the body part retargeting strategy and PAN. Our code is publicly available.
- 3 June 2023
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (9) , 2356-2376
Towards Fine-Grained Optimal 3D Face Dense Registration: An Iterative Dividing and Diffusing Method
- 3 June 2023
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (9) , 2356-2376
Towards Fine-Grained Optimal 3D Face Dense Registration: An Iterative Dividing and Diffusing Method
- 19 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 30 (8) , 4792-4808
Pose-Aware Attention Network for Flexible Motion Retargeting by Body Part
Motion retargeting is a fundamental problem in computer graphics and computer vision. Existing approaches usually have many strict requirements, such as the source-target skeletons needing to have the same number of joints or share the same topology. To tackle this problem, we note that skeletons with different structure may have some common body parts despite the differences in joint numbers. Following this observation, we propose a novel, flexible motion retargeting framework. The key idea of our method is to regard the body part as the basic retargeting unit rather than directly retargeting the whole body motion. To enhance the spatial modeling capability of the motion encoder, we introduce a pose-aware attention network (PAN) in the motion encoding phase. The PAN is pose-aware since it can dynamically predict the joint weights within each body part based on the input pose, and then construct a shared latent space for each body part by feature pooling. Extensive experiments show that our approach can generate better motion retargeting results both qualitatively and quantitatively than state-of-the-art methods. Moreover, we also show that our framework can generate reasonable results even for a more challenging retargeting scenario, like retargeting between bipedal and quadrupedal skeletons because of the body part retargeting strategy and PAN.
- 19 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 30 (8) , 4792-4808
Pose-Aware Attention Network for Flexible Motion Retargeting by Body Part
Motion retargeting is a fundamental problem in computer graphics and computer vision. Existing approaches usually have many strict requirements, such as the source-target skeletons needing to have the same number of joints or share the same topology. To tackle this problem, we note that skeletons with different structure may have some common body parts despite the differences in joint numbers. Following this observation, we propose a novel, flexible motion retargeting framework. The key idea of our method is to regard the body part as the basic retargeting unit rather than directly retargeting the whole body motion. To enhance the spatial modeling capability of the motion encoder, we introduce a pose-aware attention network (PAN) in the motion encoding phase. The PAN is pose-aware since it can dynamically predict the joint weights within each body part based on the input pose, and then construct a shared latent space for each body part by feature pooling. Extensive experiments show that our approach can generate better motion retargeting results both qualitatively and quantitatively than state-of-the-art methods. Moreover, we also show that our framework can generate reasonable results even for a more challenging retargeting scenario, like retargeting between bipedal and quadrupedal skeletons because of the body part retargeting strategy and PAN.
- 1 December 2022
- Published by Wiley in Computer Graphics Forum
- Vol. 41 (8) , 183-194
Neural3Points: Learning to Generate Physically Realistic Full‐body Motion for Virtual Reality Users
Animating an avatar that reflects a user's action in the VR world enables natural interactions with the virtual environment. It has the potential to allow remote users to communicate and collaborate in a way as if they met in person. However, a typical VR system provides only a very sparse set of up to three positional sensors, including a head‐mounted display (HMD) and Optionally two hand‐held controllers, making the estimation of the user's full‐body movement a difficult problem. In this work, we present a data‐driven physics‐based method for predicting the realistic full‐body movement of the user according to the transformations of these VR trackers and simulating an avatar character to mimic such user actions in the virtual world in realtime. We train our system using reinforcement learning with carefully designed pretraining processes to ensure the success of the training and the quality of the simulation. We demonstrate the effectiveness of the method with an extensive set of examples.
- 1 December 2022
- Published by Wiley in Computer Graphics Forum
- Vol. 41 (8) , 183-194
Neural3Points: Learning to Generate Physically Realistic Full‐body Motion for Virtual Reality Users
Animating an avatar that reflects a user's action in the VR world enables natural interactions with the virtual environment. It has the potential to allow remote users to communicate and collaborate in a way as if they met in person. However, a typical VR system provides only a very sparse set of up to three positional sensors, including a head‐mounted display (HMD) and Optionally two hand‐held controllers, making the estimation of the user's full‐body movement a difficult problem. In this work, we present a data‐driven physics‐based method for predicting the realistic full‐body movement of the user according to the transformations of these VR trackers and simulating an avatar character to mimic such user actions in the virtual world in realtime. We train our system using reinforcement learning with carefully designed pretraining processes to ensure the success of the training and the quality of the simulation. We demonstrate the effectiveness of the method with an extensive set of examples.