Xiaokun Dai
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Xiaokun Dai has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Xinrong Chen
Fudan University
4 shared publications - Xinhan Di
Independent Researcher China
2 shared publications Xinkang Zhang
Fudan University, Shanghai, China
2 shared publications- Ziqun Zhang
Fudan University
2 shared publications - Fang Li
Shanghai University of Electric Power, Shanghai, China
1 shared publication - Fuming Yang
Department of Neurosurgery, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
1 shared publication
Latest Publications
- 25 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 5029-5037
Enhancing Motion Reconstruction From Sparse Tracking Inputs With Kinematic Constraints
In virtual reality, there is a growing demand for reconstructing accurate full-body 3D avatars from the sparse motion captured through head-mounted displays and hand-held controllers. However, due to the limited information from sparse inputs, precisely reconstructing full body poses is an ill-posed and challenging task. Existing methods often exhibit notable errors in lower body poses, which results in unrealistic poses and occasional floor penetration artifacts. To address the above issue, a MLP-based model with Kinematic Constraints and Temporal Diversity (KCTD) was proposed for full body poses reconstruction, which incorporates Kinematic Constraints Hierarchical Decoder with Temporal Diversity Awareness Module and a Generative Feedback Module to further improve the accuracy of the reconstruction of the full body poses. Specifically, the potential constraints of human kinematic chain are incorporated into the model through a hierarchical decoder, which elevates overall precision through the interaction of the human kinematic chain. Then, a temporal diversity awareness module is integrated into the hierarchical decoder to help the model capture information at different frequency in the time domain. In addition, a generative feedback module is imposed on leg poses reconstruction to further improve its accuracy without increasing the model’s inference time. Test results on the AMASS dataset demonstrate that, the proposed model effectively improves the reconstruction accuracy of the full-body poses with the mean per joint rotation error and position error of of $2.60$ and $3.62$ respectively, which surpasses the state-of-the-art methods. Particularly, the proposed model can alleviate irrational poses in the lower body and reduce the floor penetration artifacts. Note to Practitioners —The motivation behind this manuscript stems from the necessity for precise full pose reconstruction in certain virtual reality applications. While existing methods have significantly contributed to the reconstruction of full-body poses based on sparse inputs, they have not taken tailored measures to improve accuracy in the lower body. Consequently, this paper introduces a full human body pose reconstruction model grounded in constraints from the human kinematic chain, featuring a temporal diversity awareness module. This model excels in reconstructing more accurate complete human body poses using sparse inputs from the head and hands, notably addressing issues such as irrational lower-body poses and floor-penetrating artifacts. Benefiting from state-of-the-art precision, the proposed approach has the potential to significantly elevate user experience and immersion in virtual reality applications, particularly in scenarios such as automated task training and simulation of hazardous conditions.
- 25 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 5029-5037
Enhancing Motion Reconstruction From Sparse Tracking Inputs With Kinematic Constraints
In virtual reality, there is a growing demand for reconstructing accurate full-body 3D avatars from the sparse motion captured through head-mounted displays and hand-held controllers. However, due to the limited information from sparse inputs, precisely reconstructing full body poses is an ill-posed and challenging task. Existing methods often exhibit notable errors in lower body poses, which results in unrealistic poses and occasional floor penetration artifacts. To address the above issue, a MLP-based model with Kinematic Constraints and Temporal Diversity (KCTD) was proposed for full body poses reconstruction, which incorporates Kinematic Constraints Hierarchical Decoder with Temporal Diversity Awareness Module and a Generative Feedback Module to further improve the accuracy of the reconstruction of the full body poses. Specifically, the potential constraints of human kinematic chain are incorporated into the model through a hierarchical decoder, which elevates overall precision through the interaction of the human kinematic chain. Then, a temporal diversity awareness module is integrated into the hierarchical decoder to help the model capture information at different frequency in the time domain. In addition, a generative feedback module is imposed on leg poses reconstruction to further improve its accuracy without increasing the model’s inference time. Test results on the AMASS dataset demonstrate that, the proposed model effectively improves the reconstruction accuracy of the full-body poses with the mean per joint rotation error and position error of of $2.60$ and $3.62$ respectively, which surpasses the state-of-the-art methods. Particularly, the proposed model can alleviate irrational poses in the lower body and reduce the floor penetration artifacts. Note to Practitioners —The motivation behind this manuscript stems from the necessity for precise full pose reconstruction in certain virtual reality applications. While existing methods have significantly contributed to the reconstruction of full-body poses based on sparse inputs, they have not taken tailored measures to improve accuracy in the lower body. Consequently, this paper introduces a full human body pose reconstruction model grounded in constraints from the human kinematic chain, featuring a temporal diversity awareness module. This model excels in reconstructing more accurate complete human body poses using sparse inputs from the head and hands, notably addressing issues such as irrational lower-body poses and floor-penetrating artifacts. Benefiting from state-of-the-art precision, the proposed approach has the potential to significantly elevate user experience and immersion in virtual reality applications, particularly in scenarios such as automated task training and simulation of hazardous conditions.
- 29 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (9) , 11242-11251
Hand-Object Pose Estimation and Reconstruction Based on Signed Distance Field and Multiscale Feature Interaction
The study of reconstruction of hands and objects from color monocular images has garnered considerable attention in recent years. In existing methods, parametric models are constructed at single scale, and the interaction between hands and objects has not fully be explored. As a result, the multiscale information in 2D images cannot be fully exploited. At the same time, the lack of feature fusion and insufficient utilization of labels also have a great impact on the reconstruction accuracy. To address the limitations, a new framework is proposed, which comprises three key modules. Firstly, a multiscale feature extractor, which generates a multiscale representation of feature, is used to capture the interaction between hand and object more effectively. Secondly, a bridge based on attention has been used to establish the connection between hand and object representations, which facilitates the integration of them. Lastly, a module based on token merge is introduced into the framework, which provides the segmentation representation of object. The experimental results on two datasets, named Obman and DexYCB, demonstrated that the proposed method had good performance and achieved a shape error about 0.121 $\text{cm}^{2}$ on Obman and 0.40 $\text{cm}^{2}$ on DexYCB, outperforming the state-of-the-art methods. This study will probably provide the human-computer interaction methods with broader application prospects.
- 29 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (9) , 11242-11251
Hand-Object Pose Estimation and Reconstruction Based on Signed Distance Field and Multiscale Feature Interaction
The study of reconstruction of hands and objects from color monocular images has garnered considerable attention in recent years. In existing methods, parametric models are constructed at single scale, and the interaction between hands and objects has not fully be explored. As a result, the multiscale information in 2D images cannot be fully exploited. At the same time, the lack of feature fusion and insufficient utilization of labels also have a great impact on the reconstruction accuracy. To address the limitations, a new framework is proposed, which comprises three key modules. Firstly, a multiscale feature extractor, which generates a multiscale representation of feature, is used to capture the interaction between hand and object more effectively. Secondly, a bridge based on attention has been used to establish the connection between hand and object representations, which facilitates the integration of them. Lastly, a module based on token merge is introduced into the framework, which provides the segmentation representation of object. The experimental results on two datasets, named Obman and DexYCB, demonstrated that the proposed method had good performance and achieved a shape error about 0.121 $\text{cm}^{2}$ on Obman and 0.40 $\text{cm}^{2}$ on DexYCB, outperforming the state-of-the-art methods. This study will probably provide the human-computer interaction methods with broader application prospects.
- 26 October 2023
- Published by World Scientific Pub Co Pte Ltd in Journal of Mechanics in Medicine and Biology
- Vol. 23 (09)
AUGMENTED REALITY-BASED NASAL ENDOSCOPE VIDEO RECONSTRUCTION AND REGISTRATION
Neuroendoscopic surgery is a minimally invasive surgical technique commonly used to remove a tumor through the patient’s mouth or nose, which requires the surgeon to avoid important neural and vascular structures. Augmented reality-based surgical navigation can provide surgeons with more information and visual aids, clarifying the structure and details of the surgical area. It requires real-time registration of intraoperative endoscopic video with pre-operative CT models. However, 3D reconstruction and camera tracking from nasal endoscopic video are challenging due to the narrow nasal cavity and the lack of texture. Besides, the nasal endoscopic datasets that can be used for deep learning-based depth estimation are scarce and hard to elaborate. To this end, a style-augmented module (SAM) is proposed in this study to minimize discrepancies between different endoscopic datasets. An unsupervised approach was trained to generate depth maps and camera motion paths, which were used to reconstruct the nasal scene’s 3D model. The obtained reconstruction was aligned with a pre-operative CT model for augmented reality-based surgical navigation. The results showed that the proposed SAM improved the generalization of the depth estimation model on nasal endoscopy data. The proposed approach to combining augmented reality technology with surgical navigation is considered instrumental in furnishing surgeons with richer information on the nasal endoscopic surgical scene.
- 26 October 2023
- Published by World Scientific Pub Co Pte Ltd in Journal of Mechanics in Medicine and Biology
- Vol. 23 (09)
AUGMENTED REALITY-BASED NASAL ENDOSCOPE VIDEO RECONSTRUCTION AND REGISTRATION
Neuroendoscopic surgery is a minimally invasive surgical technique commonly used to remove a tumor through the patient’s mouth or nose, which requires the surgeon to avoid important neural and vascular structures. Augmented reality-based surgical navigation can provide surgeons with more information and visual aids, clarifying the structure and details of the surgical area. It requires real-time registration of intraoperative endoscopic video with pre-operative CT models. However, 3D reconstruction and camera tracking from nasal endoscopic video are challenging due to the narrow nasal cavity and the lack of texture. Besides, the nasal endoscopic datasets that can be used for deep learning-based depth estimation are scarce and hard to elaborate. To this end, a style-augmented module (SAM) is proposed in this study to minimize discrepancies between different endoscopic datasets. An unsupervised approach was trained to generate depth maps and camera motion paths, which were used to reconstruct the nasal scene’s 3D model. The obtained reconstruction was aligned with a pre-operative CT model for augmented reality-based surgical navigation. The results showed that the proposed SAM improved the generalization of the depth estimation model on nasal endoscopy data. The proposed approach to combining augmented reality technology with surgical navigation is considered instrumental in furnishing surgeons with richer information on the nasal endoscopic surgical scene.
- 7 October 2023
- Published by World Scientific Pub Co Pte Ltd in Journal of Mechanics in Medicine and Biology
- Vol. 23 (09)
ISC-TRANSUNET: MEDICAL IMAGE SEGMENTATION NETWORK BASED ON THE INTEGRATION OF SELF-ATTENTION AND CONVOLUTION
In the current medical image segmentation network, the combination of CNN and Transformer has become a mainstream trend. However, the inherent limitations of convolution operation in CNN and insufficient information interaction in Transformer affect the segmentation performance of the network. To solve these problems, an integrated self-attention and convolution medical image segmentation network (ISC-TransUNet) is proposed in this paper. The network consists of encoder, decoder and jump connection. First, the encoder uses a hybrid structure of BoTNet and Transformer to capture more comprehensive image information and reduce additional computing overhead. Then, the decoder uses an upper sampler cascaded by multiple DUpsampling upper blocks to accurately recover the pixel-level prediction. Finally, the feature fusion of encoder and decoder at different resolutions is realized by ResPath jump connection, which reduces the semantic difference between encoder and decoder. Through experiments on the Synapse multi-organ segmentation dataset, compared with the baseline model TransUNet, Dice similarity coefficient of ISC-TransUNet was improved by 1.13%, Hausdorff distance was reduced by 2.38%, and weight was maintained. The experimental results show that the network can effectively segment tissues and organs in medical images, which has important theoretical significance and application value for intelligent clinical diagnosis and treatment.
- 7 October 2023
- Published by World Scientific Pub Co Pte Ltd in Journal of Mechanics in Medicine and Biology
- Vol. 23 (09)
ISC-TRANSUNET: MEDICAL IMAGE SEGMENTATION NETWORK BASED ON THE INTEGRATION OF SELF-ATTENTION AND CONVOLUTION
In the current medical image segmentation network, the combination of CNN and Transformer has become a mainstream trend. However, the inherent limitations of convolution operation in CNN and insufficient information interaction in Transformer affect the segmentation performance of the network. To solve these problems, an integrated self-attention and convolution medical image segmentation network (ISC-TransUNet) is proposed in this paper. The network consists of encoder, decoder and jump connection. First, the encoder uses a hybrid structure of BoTNet and Transformer to capture more comprehensive image information and reduce additional computing overhead. Then, the decoder uses an upper sampler cascaded by multiple DUpsampling upper blocks to accurately recover the pixel-level prediction. Finally, the feature fusion of encoder and decoder at different resolutions is realized by ResPath jump connection, which reduces the semantic difference between encoder and decoder. Through experiments on the Synapse multi-organ segmentation dataset, compared with the baseline model TransUNet, Dice similarity coefficient of ISC-TransUNet was improved by 1.13%, Hausdorff distance was reduced by 2.38%, and weight was maintained. The experimental results show that the network can effectively segment tissues and organs in medical images, which has important theoretical significance and application value for intelligent clinical diagnosis and treatment.
- 1 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 675-676
An Attention-Based Signed Distance Field Estimation Method for Hand-Object Reconstruction
Joint reconstruction of hands and objects from monocular RGB images is a challenging task. In this work, we present a novel hybrid model for joint reconstruction of hands and objects. The model proposed consists of three key modules. Among them, multi-scale attention feature extractor is designed to enhance cross-scale in-formation extraction. Attention-based graph encoder can encode the graph information of the hands. Interacting attention module is applied to fuse information between hands and object. Test re-sults on ObMan dataset [6] show that our method outperforms the state-of-the-art method around 13.97% and 15.75% in $\mathrm{H}_{\text{se}}$ and $\mathrm{H}_{\text{je}}$ respectively.- 1 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 675-676
An Attention-Based Signed Distance Field Estimation Method for Hand-Object Reconstruction
Joint reconstruction of hands and objects from monocular RGB images is a challenging task. In this work, we present a novel hybrid model for joint reconstruction of hands and objects. The model proposed consists of three key modules. Among them, multi-scale attention feature extractor is designed to enhance cross-scale in-formation extraction. Attention-based graph encoder can encode the graph information of the hands. Interacting attention module is applied to fuse information between hands and object. Test re-sults on ObMan dataset [6] show that our method outperforms the state-of-the-art method around 13.97% and 15.75% in $\mathrm{H}_{\text{se}}$ and $\mathrm{H}_{\text{je}}$ respectively.