Gudrun Klinker

Clinical workflows are inherently complex, and evaluating future technologies, e.g., the involvement of a humanoid robot, with patients is challenging. A bridge is needed when transitioning from the laboratory to the hospital. Immersive Virtual Environments (IVEs), a proven strategy for in-silico training scenarios, can ensure integration of technologies into existing care workflows, without the need for physical patients. We developed an authentic IVE in Unity that mimics a specific European hospital, including lifelike patient agents driven by a Large Language Model (LLM). Based on the findings of a pilot study, the focus was placed on including trigger-based physical props and integrating multi-user functionality. In a mixed 2x2 study design, 32 clinic staff performed an in-silico wound care scenario, providing care to the LLM-based patient. This main study is divided into two parts: Study I - the use of physical props vs. fully virtual objects; Study II - the collaborative work with another person (multi-user) vs. performing the task alone. We measured usability and presence to evaluate the physical props and the multi-user experience. These real-world results have been further analyzed, focusing on pre-existing knowledge in wound care management. Lastly, we compare the impact of the individual features overall on the simulation. Findings are then placed in context using existing studies from related publications. Our findings include: physical props contribute to presence and, through familiarity, also provide an entry point to the simulation. However, a personal selection of items as well as high fidelity is essential. Multi-user enhances presence but adds complexity to achieve the same level of task fidelity, suggesting the need for an initial training session. Other remarks include a promising future perspective on LLM-based patients, with an emphasis on situational awareness (following the action), for which more research will be necessary.

In mixed reality (MR) telepresence applications, the differences between participants' physical environments can interfere with effective collaboration. For asymmetric tasks, users might need to access different resources (information, objects, tools) distributed throughout their room. Existing intersection methods do not support such interactions, because a large portion of the telepresence participants' rooms become inaccessible, along with the relevant task resources. We propose MRUnion, a Mixed Reality Telepresence pipeline for asymmetric task-aware 3D mutual scene generation. The key concept of our approach is to enable a user in an asymmetric telecollaboration scenario to access the entire room, while still being able to communicate with remote users in a shared space. For this purpose, we introduce a novel mutual room layout called Union. We evaluated 882 space combinations quantitatively involving two, three, and four combined remote spaces and compared it to a conventional Intersect room layout. The results show that our method outperforms existing intersection methods and enables a significant increase in space and accessibility to resources within the shared space. In an exploratory user study (N=24), we investigated the applicability of the synthetic mutual scene in both MR and VR setups, where users collaborated on an asymmetric remote assembly task. The study results showed that our method achieved comparable results to the intersect method but requires further investigation in terms of social presence, safety and support of collaboration. From this study, we derived design implications for synthetic mutual spaces.

With many contemporary video conferencing platforms available, there is still a need for platforms that afford a researcher workflow to conduct controlled online experiments. We have developed an open source experimental video conferencing platform that enables researchers to design and conduct remote experiments. Our platform provides a high level of control over the user interface and video streams, which is essential for studying the differences between remote and in-person social interactions. We give an overview of our platform’s usage and architecture and conduct a take-home study (N=9) to evaluate how accessible our system is to potential new contributors. We also follow up with an initial evaluation of technical performance bottlenecks for when our experimental platform is deployed, and show that the computational resources increases per each video stream as well as the type of filters applied to each participant. We end with a short discussion on next steps and the experimental hub’s potential to be extended as a sandbox for testing browser based augmented reality (WebAR) filters to be adopted in interdisciplinary experimental procedures.