Daniel Roth

Clinical workflows are inherently complex, and evaluating future technologies, e.g., the involvement of a humanoid robot, with patients is challenging. A bridge is needed when transitioning from the laboratory to the hospital. Immersive Virtual Environments (IVEs), a proven strategy for in-silico training scenarios, can ensure integration of technologies into existing care workflows, without the need for physical patients. We developed an authentic IVE in Unity that mimics a specific European hospital, including lifelike patient agents driven by a Large Language Model (LLM). Based on the findings of a pilot study, the focus was placed on including trigger-based physical props and integrating multi-user functionality. In a mixed 2x2 study design, 32 clinic staff performed an in-silico wound care scenario, providing care to the LLM-based patient. This main study is divided into two parts: Study I - the use of physical props vs. fully virtual objects; Study II - the collaborative work with another person (multi-user) vs. performing the task alone. We measured usability and presence to evaluate the physical props and the multi-user experience. These real-world results have been further analyzed, focusing on pre-existing knowledge in wound care management. Lastly, we compare the impact of the individual features overall on the simulation. Findings are then placed in context using existing studies from related publications. Our findings include: physical props contribute to presence and, through familiarity, also provide an entry point to the simulation. However, a personal selection of items as well as high fidelity is essential. Multi-user enhances presence but adds complexity to achieve the same level of task fidelity, suggesting the need for an initial training session. Other remarks include a promising future perspective on LLM-based patients, with an emphasis on situational awareness (following the action), for which more research will be necessary.

Handling manual assembly tasks is a crucial part of everyday life. Especially with surgical instruments, proficient handling is critical for flawless workflows, efficiency, and overall patient safety. Markerless object pose estimation and augmented reality (AR) visualization can guide users through the assembly process. There is a lack of understanding when combining state-of-the-art object tracking with different types of AR assembly visualizations. Our work introduces a comparison between tabletop in-situ, FoV-fixed, and world-fixed visual guidance techniques. We use a benchmark task with step-by-step assembly guidance via gaze control and reproducible 3D-printed assemblies of varying complexity and shape (proxies for surgical instruments). Our within-subject user study with 32 participants (16 scrub nurses and 16 laypersons) revealed that AR assembly guidance can match and partially outperform paper-based instructions regarding usability, user experience, and workload. Objective metrics (times and errors) were largely comparable to the paper baseline. We identified that world-fixed and FoV-fixed animations tend to overall outperform in-situ animations, which by design require sequential visual processing and whose augmentations’ precision is most affected by pose estimation inaccuracies. We also contrast characteristics of expert and layperson feedback. All participants commended the gaze-driven testbed interface for step-by-step AR instructions, but also revealed needs for further optimization and individualization. Our insights on visualization placement characteristics using pose estimation and involving a spectrum of users and assemblies open a new perspective in the field and aim to foster future work towards pose estimation-driven AR assembly guidance.

Cross Reality (CR) is a new emerging field based on the current developments in Mixed Reality hardware, especially supported by the broad market penetration of video-based see-through Head-Mounted Displays. It refers to applications that span across different stages (real, Augmented Reality, Augmented Virtuality, Virtual Reality) of the reality-virtuality continuum, where users are interconnected between different stages and/or are able to transition between these stages. This publication follows the concept of other grand challenges publications and reflects the discussion of various researchers invested in CR. After an initial discussion at the 1^st Joint Workshop on Cross Reality at IEEE ISMAR 2023, six topic groups have been identified, leading to 22 challenges, which were discussed in groups over the period of multiple months. The discussion of these challenges should act as a road map for future research in the area of CR.

A convincing sense of embodiment in virtual reality (VR) is crucial for creating immersive and engaging experiences, as it shapes how users perceive and interact with their virtual bodies. The sense of embodiment is thereby, among others, affected by the shape, appearance, and fidelity of the virtual body. However, achieving convincing avatar appearance remains a challenge for VR applications. One promising solution is Pass-Through Embodiment (PTE), which enables users to see their real bodies in VR. PTE combines depth-based segmentation with the pass-through video stream of video-see-through displays to effectively visualize photon-captured representations of their own bodies. Despite the source-fidelity of the representation, the resolution of integrated depth sensors in Head Mounted Displays (HMD) can produce artifacts at segmentation boundaries, leading to visible aliasing. The perceptual impact of these artifacts on the VR experience remains unexplored. Therefore, in this paper we compare three edge-rendering techniques designed to reduce artifacts without compromising performance. Aside from a soft gradient, we introduce two new methods with a hard and dithered edge. The latter aims to balance the sharpness of hard masks with the smoothness of gradient transitions, without relying on alpha blending. To evaluate those methods in a PTE context, we conducted a within-subjects study that introduces a novel real-mirror paradigm, using an actual physical mirror as reference for reflection. We found significant results in measured presence and embodiment. Subsequent analysis revealed that our dithered cutout approach significantly outperforms hard masks, while no significant difference was found between soft condition. These results suggest a perceptual continuum where dithering and soft blending both effectively reduce visual artifacts through gradient representation. Together with high overall ratings on presence and embodiment across all conditions, these findings confirm PTE as a robust method for supporting embodiment and presence, while highlighting the potential of dithering as a computationally efficient yet perceptually comparable alternative to smooth blending.

Immersive virtual reality learning environments (IVRLEs) are increasingly used in medical education, yet the role of environmental fidelity—particularly scene design—remains underexplored. This study examines how varying levels of fidelity and contextualization affect motivational and cognitive outcomes. Eighty-seven medical students were randomly assigned to one of three scene conditions: a minimalistic “Blank Scene,” a “Reconstructed Classroom Scene”, or an “Inside-Human Scene”. All students used a custom-developed application to learn about embryonic heart development. We measured virtual presence, intrinsic motivation, cognitive load, learning outcomes, and usability. Results showed that scene design influenced virtual presence, selected aspects of intrinsic motivation, cognitive load, and learning outcomes. The Inside-Human Scene elicited higher physical and self-presence as well as higher comprehension scores compared to the Reconstructed Classroom Scene. The Reconstructed Classroom Scene was associated with higher extraneous cognitive load. Intrinsic cognitive load was rated higher in the Inside-Human Scene, while germane cognitive load did not differ between conditions. No significant differences were found for task performance or factual recall. Overall, the findings indicate that scene design in IVRLEs affects how learners engage with complex content and may support deeper understanding when perceptual and contextual properties are coherent, while visually detailed environments may increase extraneous cognitive demands without improving learning.

Multi-camera dynamic Augmented Reality (AR) applications require a camera pose estimation to leverage individual information from each camera in one common system. This can be achieved by combining contextual information, such as markers or objects, across multiple views. While commonly cameras are calibrated in an initial step or updated through the constant use of markers, another option is to leverage information already present in the scene, like known objects. Another downside of marker-based tracking is that markers have to be tracked inside the field-of-view (FoV) of the cameras. To overcome these limitations, we propose a constant dynamic camera pose estimation leveraging spatiotemporal FoV overlaps of known objects on the fly. To achieve that, we enhance the state-of-the-art object pose estimator to update our spatiotemporal scene graph, enabling a relation even among non-overlapping FoV cameras. To evaluate our approach, we introduce a multi-camera, multi-object pose estimation dataset with temporal FoV overlap, including static and dynamic cameras. Furthermore, in FoV overlapping scenarios, we outperform the state-of-the-art on the widely used YCB-V and T-LESS dataset in camera pose accuracy. Our performance on both previous and our proposed datasets validates the effectiveness of our marker-less approach for AR applications. The code and dataset are available on https://github.com/roth-hex-lab/IEEE-VR-2026-MultiCam.