Jingying Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jingying Wang has a total of 33 co-authors. Here are the top co-authors by number of shared publications:
- Vitaliy Popov
University of Michigan Medical School
4 shared publications - Xu Wang
University of Michigan, Ann Arbor, MI, USA
4 shared publications - Taylor Kantor
Department of Surgery, Michigan Medicine, United States
2 shared publications - Chen Chen
Florida International University, Miami, USA
1 shared publication - Chun Yu
Department of Computer Science and Technology, Tsinghua University, Beijing, China
1 shared publication - Gurjit Sandhu
Department of Surgery, Michigan Medicine, United States
1 shared publication
Latest Publications
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-5
From What Is in Textbooks to What Is in Experts: Human-AI Systems for Externalizing Surgical Expertise
Surgical expertise spans more than what textbooks can include. Trainees must acquire explicit knowledge, such as operative decision-making; psychomotor skills accumulated through thousands of hours of practice; and attentional skills, such as knowing what to attend to, which even experts often struggle to articulate. Current training resources address the explicit level well but offer little support for these tacit levels, while access to expert time in the operating room continues to shrink. My dissertation develops human–AI systems that externalize surgical expertise across these three increasingly tacit levels. Surgment and SurgGraph use computer vision to turn surgical recordings into visual exercises with feedback. eXplainMR teaches ultrasonography in mixed reality, explaining why each movement is recommended to help trainees build three-dimensional mental models. SurgGaze enables accurate gaze capture in the operating room through implicit calibration, revealing how experts allocate attention, including strategies such as focus and anticipation. I plan to integrate these levels into residency training through post-operative debriefing that reviews decision-making, action efficiency, and attention allocation in one loop.
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Open AccessFrom What Is in Textbooks to What Is in Experts: Human-AI Systems for Externalizing Surgical Expertise
Surgical expertise spans more than what textbooks can include. Trainees must acquire explicit knowledge, such as operative decision-making; psychomotor skills accumulated through thousands of hours of practice; and attentional skills, such as knowing what to attend to, which even experts often struggle to articulate. Current training resources address the explicit level well but offer little support for these tacit levels, while access to expert time in the operating room continues to shrink. My dissertation develops human–AI systems that externalize surgical expertise across these three increasingly tacit levels. Surgment and SurgGraph use computer vision to turn surgical recordings into visual exercises with feedback. eXplainMR teaches ultrasonography in mixed reality, explaining why each movement is recommended to help trainees build three-dimensional mental models. SurgGaze enables accurate gaze capture in the operating room through implicit calibration, revealing how experts allocate attention, including strategies such as focus and anticipation. I plan to integrate these levels into residency training through post-operative debriefing that reviews decision-making, action efficiency, and attention allocation in one loop.
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
ReadGaze: Query the Gaze, Shape the Debrief
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessReadGaze: Query the Gaze, Shape the Debrief
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-18
eXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Open AccesseXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR
- Open Access
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Surgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery Learning
Videos are prominent learning materials to prepare surgical trainees before they enter the operating room (OR). In this work, we explore techniques to enrich the video-based surgery learning experience. We propose Surgment, a system that helps expert surgeons create exercises with feedback based on surgery recordings. Surgment is powered by a few-shot-learning-based pipeline (SegGPT+SAM) to segment surgery scenes, achieving an accuracy of 92%. The segmentation pipeline enables functionalities to create visual questions and feedback desired by surgeons from a formative study. Surgment enables surgeons to 1) retrieve frames of interest through sketches, and 2) design exercises that target specific anatomical components and offer visual feedback. In an evaluation study with 11 surgeons, participants applauded the search-by-sketch approach for identifying frames of interest and found the resulting image-based questions and feedback to be of high educational value.
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Open AccessSurgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery Learning
Videos are prominent learning materials to prepare surgical trainees before they enter the operating room (OR). In this work, we explore techniques to enrich the video-based surgery learning experience. We propose Surgment, a system that helps expert surgeons create exercises with feedback based on surgery recordings. Surgment is powered by a few-shot-learning-based pipeline (SegGPT+SAM) to segment surgery scenes, achieving an accuracy of 92%. The segmentation pipeline enables functionalities to create visual questions and feedback desired by surgeons from a formative study. Surgment enables surgeons to 1) retrieve frames of interest through sketches, and 2) design exercises that target specific anatomical components and offer visual feedback. In an evaluation study with 11 surgeons, participants applauded the search-by-sketch approach for identifying frames of interest and found the resulting image-based questions and feedback to be of high educational value.
- Open Access
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Looking Together ≠ Seeing the Same Thing: Understanding Surgeons' Visual Needs During Intra-operative Coordination and Instruction
Shared gaze visualizations have been found to enhance collaboration and communication outcomes in diverse HCI scenarios including computer supported collaborative work and learning contexts. Given the importance of gaze in surgery operations, especially when a surgeon trainer and trainee need to coordinate their actions, research on the use of gaze to facilitate intra-operative coordination and instruction has been limited and shows mixed implications. We performed a field observation of 8 surgeries and an interview study with 14 surgeons to understand their visual needs during operations, informing ways to leverage and augment gaze to enhance intra-operative coordination and instruction. We found that trainees have varying needs in receiving visual guidance which are often unfulfilled by the trainers’ instructions. It is critical for surgeons to control the timing of the gaze-based visualizations and effectively interpret gaze data. We suggest overlay technologies, e.g., gaze-based summaries and depth sensing, to augment raw gaze in support of surgical coordination and instruction.
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessLooking Together ≠ Seeing the Same Thing: Understanding Surgeons' Visual Needs During Intra-operative Coordination and Instruction
Shared gaze visualizations have been found to enhance collaboration and communication outcomes in diverse HCI scenarios including computer supported collaborative work and learning contexts. Given the importance of gaze in surgery operations, especially when a surgeon trainer and trainee need to coordinate their actions, research on the use of gaze to facilitate intra-operative coordination and instruction has been limited and shows mixed implications. We performed a field observation of 8 surgeries and an interview study with 14 surgeons to understand their visual needs during operations, informing ways to leverage and augment gaze to enhance intra-operative coordination and instruction. We found that trainees have varying needs in receiving visual guidance which are often unfulfilled by the trainers’ instructions. It is critical for surgeons to control the timing of the gaze-based visualizations and effectively interpret gaze data. We suggest overlay technologies, e.g., gaze-based summaries and depth sensing, to augment raw gaze in support of surgical coordination and instruction.
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-3
SketchSearch: Fine-tuning Reference Maps to Create Exercises In Support of Video-based Learning for Surgeons
Video-based surgical coaching involves mentors reviewing surgery footage with trainees. Although effective, its use is sporadic due to time constraints. We propose AI-augmented coaching through SketchSearch, allowing experts to create exercises with automated feedback in surgery videos for self-learning. Surgeons often seek specific scenes for teaching, relying on visual cues. SketchSearch simplifies this through a three-step process: key frame extraction, template reference maps creation via image segmentation, and fine-tuning for frame retrieval.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessSketchSearch: Fine-tuning Reference Maps to Create Exercises In Support of Video-based Learning for Surgeons
Video-based surgical coaching involves mentors reviewing surgery footage with trainees. Although effective, its use is sporadic due to time constraints. We propose AI-augmented coaching through SketchSearch, allowing experts to create exercises with automated feedback in surgery videos for self-learning. Surgeons often seek specific scenes for teaching, relying on visual cues. SketchSearch simplifies this through a three-step process: key frame extraction, template reference maps creation via image segmentation, and fine-tuning for frame retrieval.
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 6851-6859
Real-time Facial Animation for 3D Stylized Character with Emotion Dynamics
Our aim is to improve animation production techniques' efficiency and effectiveness. We present two real-time solutions which drive character expressions in a geometrically consistent and perceptually valid way. Our first solution combines keyframe animation techniques with machine learning models. We propose a 3D emotion transfer network makes use of a 2D human image to generate a stylized 3D rig parameter. Our second solution combines blendshape-based motion capture animation techniques with machine learning models. We propose a blendshape adaption network which generates the character rig parameter motions with geometric consistency and temporally stability. We demonstrate the effectiveness of our system by comparing it to a commercial product Faceware. Results reveal that ratings of the recognition, intensity, and attractiveness of expressions depicted for animated characters via our systems are statistically higher than Faceware. Our results may be implemented into the animation pipeline, supporting animators to create expressions more rapidly and precisely.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 6851-6859
Open AccessReal-time Facial Animation for 3D Stylized Character with Emotion Dynamics
Our aim is to improve animation production techniques' efficiency and effectiveness. We present two real-time solutions which drive character expressions in a geometrically consistent and perceptually valid way. Our first solution combines keyframe animation techniques with machine learning models. We propose a 3D emotion transfer network makes use of a 2D human image to generate a stylized 3D rig parameter. Our second solution combines blendshape-based motion capture animation techniques with machine learning models. We propose a blendshape adaption network which generates the character rig parameter motions with geometric consistency and temporally stability. We demonstrate the effectiveness of our system by comparing it to a commercial product Faceware. Results reveal that ratings of the recognition, intensity, and attractiveness of expressions depicted for animated characters via our systems are statistically higher than Faceware. Our results may be implemented into the animation pipeline, supporting animators to create expressions more rapidly and precisely.
- 22 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 29 (5) , 2647-2657
ConeSpeech: Exploring Directional Speech Interaction for Multi-Person Remote Communication in Virtual Reality
Remote communication is essential for efficient collaboration among people at different locations. We present ConeSpeech, a virtual reality (VR) based multi-user remote communication technique, which enables users to selectively speak to target listeners without distracting bystanders. With ConeSpeech, the user looks at the target listener and only in a cone-shaped area in the direction can the listeners hear the speech. This manner alleviates the disturbance to and avoids overhearing from surrounding irrelevant people. Three featured functions are supported, directional speech delivery, size-adjustable delivery range, and multiple delivery areas, to facilitate speaking to more than one listener and to listeners spatially mixed up with bystanders. We conducted a user study to determine the modality to control the cone-shaped delivery area. Then we implemented the technique and evaluated its performance in three typical multi-user communication tasks by comparing it to two baseline methods. Results show that ConeSpeech balanced the convenience and flexibility of voice communication.
- 22 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 29 (5) , 2647-2657
ConeSpeech: Exploring Directional Speech Interaction for Multi-Person Remote Communication in Virtual Reality
Remote communication is essential for efficient collaboration among people at different locations. We present ConeSpeech, a virtual reality (VR) based multi-user remote communication technique, which enables users to selectively speak to target listeners without distracting bystanders. With ConeSpeech, the user looks at the target listener and only in a cone-shaped area in the direction can the listeners hear the speech. This manner alleviates the disturbance to and avoids overhearing from surrounding irrelevant people. Three featured functions are supported, directional speech delivery, size-adjustable delivery range, and multiple delivery areas, to facilitate speaking to more than one listener and to listeners spatially mixed up with bystanders. We conducted a user study to determine the modality to control the cone-shaped delivery area. Then we implemented the technique and evaluated its performance in three typical multi-user communication tasks by comparing it to two baseline methods. Results show that ConeSpeech balanced the convenience and flexibility of voice communication.