ARMedicalSketch: Exploring 3D Sketching for Medical Image Using True 2D-3D Interlinked Visualization and Interaction
- 2 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Human-Machine Systems
- Vol. 54 (5) , 589-598
Abstract
In traditional clinical practice, doctors often have to deal with 3D information based on 2D-displayed medical images. There is a considerable mismatch between the 2D and 3D dimensions in image interaction during clinical diagnosis, making image manipulation challenging and time-consuming. In this study, we explored 3D sketching for medical images using true 2D-3D interlinked visualization and interaction, presenting a novel AR environment named ARMedicalSketch. It supports image display enhancement preprocessing and 3D interaction tasks for original 3D medical images. Our interaction interface, based on 3D autostereoscopic display technology, provides both floating 3D display and 2D tablet display while enabling glasses-free visualization. We presented a method of 2D-3D interlinked visualization and interaction, employing synchronized projection visualization and a virtual synchronized interactive plane to establish an integrated relationship between 2D and 3D displays. Additionally, we utilized gesture sensors and a 2D touch tablet to capture the user's hand information for convenient interaction. We constructed the prototype and conducted a user study involving 23 students and 2 clinical experts. The controlled study compared our proposed system with a 2D display prototype, showing enhanced efficiency in interacting with medical images while maintaining 2D interaction accuracy, particularly in tasks involving strong 3D spatial correlation. In the future, we aim to further enhance the interaction precision and application scenarios of ARMedicalSketch.Keywords
Funding Information
- National Natural Science Foundation of China (U22A2051, 82027807, 62101305)
- National Key Research and Development Program of China (2022YFC2405200)
- Intelligent Healthcare Tsinghua University (2022ZLB001)
- Tsinghua-Foshan Innovation Special Fund (2021THFS0104)
This publication has 32 references indexed in Scilit:
- A VR Environment for Human Anatomical Variation Education: Modeling, Visualization and InteractionIEEE Transactions on Learning Technologies, 2022
- Hybrid Touch/Tangible Spatial 3D Data SelectionComputer Graphics Forum, 2019
- Three-Dimensional Visualization Technology Used in Pancreatic Surgery: a Valuable Tool for Surgical TraineesJournal of Gastrointestinal Surgery, 2019
- Holographic near-eye displays for virtual and augmented realityACM Transactions on Graphics, 2017
- Enabling Focus Cues in Head-Mounted DisplaysProceedings of the IEEE, 2017
- Sketch‐Based Editing Tools for Tumour Segmentation in 3D Medical ImagesComputer Graphics Forum, 2013
- Contour‐based Interface for Refining Volume SegmentationComputer Graphics Forum, 2010
- 3-D Augmented Reality for MRI-Guided Surgery Using Integral Videography Autostereoscopic Image OverlayIEEE Transactions on Biomedical Engineering, 2010
- Comparison of two- and three-dimensional camera systems in laparoscopic performance: a novel 3D system with one cameraSurgical Endoscopy, 2009
- Drawing on Air: Input Techniques for Controlled 3D Line IllustrationIEEE Transactions on Visualization and Computer Graphics, 2007