Zhaoshuo Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhaoshuo Li has a total of 59 co-authors. Here are the top co-authors by number of shared publications:
- Anton Deguet
Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA
2 shared publications Mathias Unberath
Johns Hopkins University, Baltimore, USA
2 shared publications- Preetham Chalasani
Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
2 shared publications Russell H. Taylor
Johns Hopkins University, Baltimore, USA
2 shared publications- Russell H. Taylor
Department of Computer Science, Johns Hopkins University
2 shared publications Alan Yuille
Johns Hopkins University
1 shared publication
Latest Publications
- Open Access
- 3 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-11
SpaceMesh: A Continuous Representation for Learning Manifold Surface Meshes
- 3 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessSpaceMesh: A Continuous Representation for Learning Manifold Surface Meshes
- Open Access
- 25 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-2
GenUSD: 3D scene generation made easy
We introduce GenUSD, an end-to-end text-to-scene generation framework that transforms natural language queries into realistic 3D scenes, including 3D objects and layouts. The process involves two main steps: 1) A Large Language Model (LLM) generates a scene layout hierarchically. It first proposes a high-level plan to decompose the scene into multiple functionally and spatially distinct subscenes. Then, for each subscene, the LLM proposes objects with detailed positions, poses, sizes, and descriptions. To manage complex object relationships and intricate scenes, we introduce object layout design meta functions as tools for the LLM. 2) A novel text-to-3D model generates each 3D object with surface meshes and high-resolution texture maps based on the LLM’s descriptions. The assembled 3D assets form the final 3D scene, represented as a Universal Scene Description (USD) format. GenUSD ensures physical plausibility by incorporating functions to prevent collisions.
- 25 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-2
Open AccessGenUSD: 3D scene generation made easy
We introduce GenUSD, an end-to-end text-to-scene generation framework that transforms natural language queries into realistic 3D scenes, including 3D objects and layouts. The process involves two main steps: 1) A Large Language Model (LLM) generates a scene layout hierarchically. It first proposes a high-level plan to decompose the scene into multiple functionally and spatially distinct subscenes. Then, for each subscene, the LLM proposes objects with detailed positions, poses, sizes, and descriptions. To manage complex object relationships and intricate scenes, we introduce object layout design meta functions as tools for the LLM. 2) A novel text-to-3D model generates each 3D object with surface meshes and high-resolution texture maps based on the LLM’s descriptions. The assembled 3D assets form the final 3D scene, represented as a Universal Scene Description (USD) format. GenUSD ensures physical plausibility by incorporating functions to prevent collisions.
- 2 June 2023
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 18 (7) , 1303-1310
TAToo: vision-based joint tracking of anatomy and tool for skull-base surgery
Purpose Tracking the 3D motion of the surgical tool and the patient anatomy is a fundamental requirement for computer-assisted skull-base surgery. The estimated motion can be used both for intra-operative guidance and for downstream skill analysis. Recovering such motion solely from surgical videos is desirable, as it is compliant with current clinical workflows and instrumentation. Methods We present Tracker of Anatomy and Tool (TAToo). TAToo jointly tracks the rigid 3D motion of the patient skull and surgical drill from stereo microscopic videos. TAToo estimates motion via an iterative optimization process in an end-to-end differentiable form. For robust tracking performance, TAToo adopts a probabilistic formulation and enforces geometric constraints on the object level. Results We validate TAToo on both simulation data, where ground truth motion is available, as well as on anthropomorphic phantom data, where optical tracking provides a strong baseline. We report sub-millimeter and millimeter inter-frame tracking accuracy for skull and drill, respectively, with rotation errors below \(1^{\circ }\). We further illustrate how TAToo may be used in a surgical navigation setting. Conclusions We present TAToo, which simultaneously tracks the surgical tool and the patient anatomy in skull-base surgery. TAToo directly predicts the motion from surgical videos, without the need of any markers. Our results show that the performance of TAToo compares favorably to competing approaches. Future work will include fine-tuning of our depth network to reach a 1 mm clinical accuracy goal desired for surgical applications in the skull base.- 2 June 2023
- Published by Springer Nature in International Journal of Computer Assisted Radiology and Surgery
- Vol. 18 (7) , 1303-1310
TAToo: vision-based joint tracking of anatomy and tool for skull-base surgery
Purpose Tracking the 3D motion of the surgical tool and the patient anatomy is a fundamental requirement for computer-assisted skull-base surgery. The estimated motion can be used both for intra-operative guidance and for downstream skill analysis. Recovering such motion solely from surgical videos is desirable, as it is compliant with current clinical workflows and instrumentation. Methods We present Tracker of Anatomy and Tool (TAToo). TAToo jointly tracks the rigid 3D motion of the patient skull and surgical drill from stereo microscopic videos. TAToo estimates motion via an iterative optimization process in an end-to-end differentiable form. For robust tracking performance, TAToo adopts a probabilistic formulation and enforces geometric constraints on the object level. Results We validate TAToo on both simulation data, where ground truth motion is available, as well as on anthropomorphic phantom data, where optical tracking provides a strong baseline. We report sub-millimeter and millimeter inter-frame tracking accuracy for skull and drill, respectively, with rotation errors below \(1^{\circ }\). We further illustrate how TAToo may be used in a surgical navigation setting. Conclusions We present TAToo, which simultaneously tracks the surgical tool and the patient anatomy in skull-base surgery. TAToo directly predicts the motion from surgical videos, without the need of any markers. Our results show that the performance of TAToo compares favorably to competing approaches. Future work will include fine-tuning of our depth network to reach a 1 mm clinical accuracy goal desired for surgical applications in the skull base.- 8 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Robotics and Bionics
- Vol. 3 (1) , 31-43
Evaluation of Hybrid Control and Palpation Assistance for Situational Awareness in Telemanipulated Task Execution
The use of intelligent feedback modalities to control and react to interaction forces during surgical procedures is an important factor in enabling safe and precise surgery. We explore the use of a model-mediated telemanipulation framework to enhance a user’s situational awareness using assistive virtual fixtures and semi-automated task execution for safe and intuitive environment interaction during robotic laparoscopic surgery. The framework allows stiffness mapping with semi-autonomous excitation, hybrid position-force control, and model updates during soft geometry contact. A 24-person study was carried out at 3 sites in simulated ablation and palpation of phantom anatomy. Compared to methods lacking intelligent feedback and guidance, the proposed framework improved task execution metrics (force regulation, completion time, path-following error) and reduced user effort.
- 8 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Robotics and Bionics
- Vol. 3 (1) , 31-43
Evaluation of Hybrid Control and Palpation Assistance for Situational Awareness in Telemanipulated Task Execution
The use of intelligent feedback modalities to control and react to interaction forces during surgical procedures is an important factor in enabling safe and precise surgery. We explore the use of a model-mediated telemanipulation framework to enhance a user’s situational awareness using assistive virtual fixtures and semi-automated task execution for safe and intuitive environment interaction during robotic laparoscopic surgery. The framework allows stiffness mapping with semi-autonomous excitation, hybrid position-force control, and model updates during soft geometry contact. A 24-person study was carried out at 3 sites in simulated ablation and palpation of phantom anatomy. Compared to methods lacking intelligent feedback and guidance, the proposed framework improved task execution metrics (force regulation, completion time, path-following error) and reduced user effort.
- 16 April 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Robotics and Bionics
- Vol. 2 (2) , 176-187
Hybrid Robot-Assisted Frameworks for Endomicroscopy Scanning in Retinal Surgeries
High-resolution real-time intraocular imaging of retina at the cellular level is very challenging due to the vulnerable and confined space within the eyeball as well as the limited availability of appropriate modalities. A probe-based confocal laser endomicroscopy (pCLE) system, can be a potential imaging modality for improved diagnosis. The ability to visualize the retina at the cellular level could provide information that may predict surgical outcomes. The adoption of intraocular pCLE scanning is currently limited due to the narrow field of view and the micron-scale range of focus. In the absence of motion compensation, physiological tremors of the surgeons hand and patient movements also contribute to the deterioration of the image quality. Therefore, an image-based hybrid control strategy is proposed to mitigate the above challenges. The proposed hybrid control strategy enables a shared control of the pCLE probe between surgeons and robots to scan the retina precisely, with the absence of hand tremors and with the advantages of an image-based auto-focus algorithm that optimizes the quality of pCLE images. The hybrid control strategy is deployed on two frameworks—cooperative and teleoperated. Better image quality, smoother motion, and reduced workload are all achieved in a statistically significant manner with the hybrid control frameworks.
- 16 April 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Robotics and Bionics
- Vol. 2 (2) , 176-187
Hybrid Robot-Assisted Frameworks for Endomicroscopy Scanning in Retinal Surgeries
High-resolution real-time intraocular imaging of retina at the cellular level is very challenging due to the vulnerable and confined space within the eyeball as well as the limited availability of appropriate modalities. A probe-based confocal laser endomicroscopy (pCLE) system, can be a potential imaging modality for improved diagnosis. The ability to visualize the retina at the cellular level could provide information that may predict surgical outcomes. The adoption of intraocular pCLE scanning is currently limited due to the narrow field of view and the micron-scale range of focus. In the absence of motion compensation, physiological tremors of the surgeons hand and patient movements also contribute to the deterioration of the image quality. Therefore, an image-based hybrid control strategy is proposed to mitigate the above challenges. The proposed hybrid control strategy enables a shared control of the pCLE probe between surgeons and robots to scan the retina precisely, with the absence of hand tremors and with the advantages of an image-based auto-focus algorithm that optimizes the quality of pCLE images. The hybrid control strategy is deployed on two frameworks—cooperative and teleoperated. Better image quality, smoother motion, and reduced workload are all achieved in a statistically significant manner with the hybrid control frameworks.
- Open Access
A Robotic 3D Perception System for Operating Room Environment Awareness
Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentation algorithm is created to enable recognition of common and salient objects/equipment and surgical activities in a da Vinci OR. Our proposed 3D semantic segmentation method has been trained and validated on a novel densely annotated dataset that has been captured from clinical scenarios. Results: The results show that our proposed architecture has acceptable registration error ($3.3\%\pm1.4\%$ of object-camera distance) and can robustly improve scene segmentation performance (mean Intersection Over Union - mIOU) for less frequently appearing classes ($\ge 0.013$) compared to a single-view method. Conclusion: We present the first dynamic multi-view perception system with a novel segmentation architecture, which can be used as a building block technology for applications such as surgical workflow analysis, automation of surgical sub-tasks and advanced guidance systems.
Open AccessA Robotic 3D Perception System for Operating Room Environment Awareness
Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentation algorithm is created to enable recognition of common and salient objects/equipment and surgical activities in a da Vinci OR. Our proposed 3D semantic segmentation method has been trained and validated on a novel densely annotated dataset that has been captured from clinical scenarios. Results: The results show that our proposed architecture has acceptable registration error ($3.3\%\pm1.4\%$ of object-camera distance) and can robustly improve scene segmentation performance (mean Intersection Over Union - mIOU) for less frequently appearing classes ($\ge 0.013$) compared to a single-view method. Conclusion: We present the first dynamic multi-view perception system with a novel segmentation architecture, which can be used as a building block technology for applications such as surgical workflow analysis, automation of surgical sub-tasks and advanced guidance systems.