Quentin Lohmeyer

Purpose: Current femoral nailing simulators face a trade-off between delivering procedural guidance and realistic haptic feedback. This study aimed to address this limitation by developing and evaluating an augmented reality (AR)-enhanced physical simulator that combines real surgical instruments with AR-based visualization. The system provides spatial feedback on fluoroscopic imaging orientation and implant placement within hidden anatomical structures, supporting self-directed learning. Methods: A hybrid simulator integrating a synthetic femur, real surgical instruments, and an AR interface on the Apple Vision Pro was developed. The AR application introduces users to fluoroscopic imaging and implant placement—covering nail insertion and alignment. After completing this AR-guided phase, 25 participants (6 experts and 19 non-experts) performed three unguided simulation runs followed by one retention session. Performance metrics included task completion time, number of simulated fluoroscopic images, and implant placement accuracy. Results: Data from 23 participants were analyzed. Task completion time decreased across simulation runs (insertion: −47.1%; alignment: −57.4%) and remained stable during retention. The number of simulated fluoroscopic images declined (insertion: −45.6%; alignment: −49.2%). Implant placement accuracy remained stable. Experts consistently performed faster than non-experts, with performance differences decreasing across simulation runs. Conclusion: The AR-enhanced physical simulator led to clear learning effects in procedural efficiency and radiographic handling while maintaining consistent implant accuracy. These findings support AR as a valuable tool to enhance accessibility, spatial learning, and realism in simulation-based surgical training, representing a further step in the ongoing shift toward simulation-based surgical training.

Background: Surgical skill assessment in orthopaedic trauma surgery still relies on subjective expert ratings, which limits consistency and scalability. While digitalization offers a path toward objective and scalable assessment, the highly manual and haptic nature of surgery makes tool use difficult to capture digitally, keeping such approaches underdeveloped. This study introduces a digital assessment framework for orthopaedic trauma training that derives digital behavioral metrics (DBM) from tracked surgical tool motion. Specifically, it investigates (1) which DBM indicate technical proficiency and (2) to what extent these DBM are capable of predicting expert-rated surgical performance. Methods: Twenty-eight participants performed 3 standardized fracture fixations on synthetic bone models of the radius, ulna, and fibula. Tool motion was captured and transformed into a digital twin from which metrics such as path length, smoothness, and task duration were derived. These metrics were statistically compared with the average Global Rating Scale (GRS) obtained from 4 experts who rated each surgical performance. (1) Correlation analysis identified skill-relevant metrics and (2) a predictive model was trained to estimate performance from DBM evaluating its accuracy against the individual expert ratings. Results: (1) Several DBM were found to be indicative of surgical performance. Measures based on tool path length and time per activity showed strong correlations with expert ratings, reaching coefficients of up to 0.6. Correlation strength varied across tools and procedures. (2) The predictive model achieved a mean absolute difference of 3.8 points from the average GRS score (scale range: 28-70), outperforming the mean interexpert difference of 4.6 points. Conclusion: DBM were identified as valid indicators of surgical skill. The study further demonstrated their predictive value, showing closer alignment with experts’ average GRS score than individual expert ratings. These findings highlight the feasibility of objective, expert-independent performance assessment in orthopaedic trauma surgery training.

Background: Stroke is a leading cause of disability, often accompanied by unilateral spatial neglect (USN), which severely impairs recovery. Traditional assessments like paper-pencil tests provide limited insights into behaviors and eye–hand coordination during real-world tasks. Advances in hand pose estimation and eye tracking in combination with augmented reality (AR) offer potential for data-driven assessments of naturalistic interactions. Objective: This proof-of-concept study presents and evaluates a multimodal behavioral tracking system that captures gaze, body, and hand movements during interactions within an AR environment. Our primary goals are to (1) validate that this system can achieve robust and accurate interaction data capture in clinical settings, (2) show that the system can reliably detect known USN behavioral patterns, and (3) explore how comprehensive data can provide new understanding of eye–hand coordination deficits in USN. Methods: We developed an AR-based assessment system using Microsoft HoloLens 2 and an external body-tracking camera to capture real-time gaze, hand, and body movements in an interactive environment. Multimodal data streams were temporally synchronized, fused, and filtered to enhance spatial accuracy and availability. Tracking performance was benchmarked against a traditional optical motion-capture system to validate reliability. In a study, 7 patients with right-brain lesions with mild to moderate USN and 8 healthy controls participated. Each performed a designed reaching task, stamping virtual sheets of paper that appeared randomly on a table. We analyzed participants’ search behavior patterns to assess attentional biases and examined gaze anchoring timing during targeted reaching motions to explore potential eye–hand coordination deficits. Results: The fusion of hand-tracking data from the HoloLens 2 and external system reduced tracking loss from 25.7% to 2.4%, with an absolute trajectory error of 3.27 cm. The system demonstrated high usability and was well accepted by patients. Data from the control group confirmed the absence of intrinsic lateral biases in the system and task design. The USN group displayed typical search behavior through ipsilesional biases in gaze direction during visual exploration (median deviation 7.46 [1.61-9.48] deg, P <.05) and longer times to find contralesional targets (median difference 1.08 [0.20-1.80] s, P =.02). Additionally, the eye–hand coordination analysis revealed lateral differences in gaze anchoring during targeted reaching motions in the USN group, with earlier fixation on contralesional targets (median difference 112 [71-146] ms, P =.02). Conclusions: The proposed AR framework provides a novel, comprehensive data-driven method for capturing interaction behavior in a controlled, yet naturalistic environment. Our results demonstrated the system’s effectiveness in measuring hallmark USN symptoms, such as gaze and head orientation biases, and highlighted its potential to complement traditional assessments by offering deep insights into torso rotation and eye–hand coordination with a high resolution and accuracy. This data-driven approach shows promise for enhancing current USN assessment practices and gaining new insights into patients’ behaviors.

Purpose: Numerous navigation devices for percutaneous, CT-guided interventions exist and are, due to their advantages, increasingly integrated into the clinical workflow. However, effective training methods to ensure safe usage are still lacking. This study compares the potential of an augmented reality (AR) training application with conventional instructions for the Cube Navigation System (CNS), hypothesizing enhanced training with AR, leading to safer clinical usage. Methods: An AR-tablet app was developed to train users puncturing with CNS. In a study, 34 medical students were divided into two groups: One trained with the AR-app, while the other used conventional instructions. After training, each participant executed 6 punctures on a phantom (204 in total) following a standardized protocol to identify and measure two potential CNS procedural user errors: (1) missing the coordinates specified and (2) altering the needle trajectory during puncture. Training performance based on train time and occurrence of procedural errors, as well as scores of User Experience Questionnaire (UEQ) for both groups, was compared. Results: Training duration was similar between the groups. However, the AR-trained participants showed a 55.1% reduced frequency of the first procedural error (p > 0.05) and a 35.1% reduced extent of the second procedural error (p < 0.01) compared to the conventionally trained participants. UEQ scores favored the AR-training in five of six categories (p < 0.05). Conclusion: The AR-app enhanced training performance and user experience over traditional methods. This suggests the potential of AR-training for navigation devices like the CNS, potentially increasing their safety, ultimately improving outcomes in percutaneous needle placements.

Background: Cardiopulmonary resuscitation (CPR) training improves CPR skills while heavily relying on feedback. The quality of feedback can vary between experts, indicating a need for data-driven feedback to support experts. The goal of this study was to investigate pose estimation, a motion detection technology, to assess individual and team CPR quality with the arm angle and chest-to-chest distance metrics. Methods: After mandatory basic life support training, 91 healthcare providers performed a simulated CPR scenario in teams. Their behaviour was simultaneously rated based on pose estimation and by experts. It was assessed if the arm was straight at the elbow, by calculating the mean arm angle, and how close the distance between the team members was during chest compressions, by calculating the chest-to-chest distance. Both pose estimation metrics were compared with the expert ratings. Results: The data-driven and expert-based ratings for the arm angle differed by 77.3%, and based on pose estimation, 13.2% of participants kept the arm straight. The chest-to-chest distance ratings by expert and by pose estimation differed by 20.7% and based on pose estimation 63.2% of participants were closer than 1 m to the team member performing compressions. Conclusions: Pose estimation-based metrics assessed learners’ arm angles in more detail and their chest-to-chest distance comparably to expert ratings. Pose estimation metrics can complement educators with additional objective detail and allow them to focus on other aspects of the simulated CPR training, increasing the training’s success and the participants’ CPR quality. Trial registration: Not applicable.