Amy Banić

We present Editing Reality, a mixed reality system that enables in-situ co-creation with generative AI directly within physical environments. Rather than treating generation as a one-shot command, the system supports embodied and iterative creation through speech, sketching, and direct manipulation, allowing users to generate, modify, erase, and retexture real-anchored virtual and reconstructed scene elements in place. Using a Research Through Design approach, we investigate how co-creation unfolds through iterative system development, a formative workshop, and expert review. From this process, we articulate a set of designerly framings that characterize in-situ co-creation as a negotiated, spatial, and temporal practice shaped by previews, accumulation, waiting, embodied evaluation, and learning the system as a spatial actor. We instantiate these ideas in a working system and report expert feedback highlighting both its creative potential and its design implications. Our work contributes a conceptual lens for understanding generative AI in mixed reality not as a one-shot automation tool, but as part of an embodied, situated creative process.

Mobile video calls are widely used to share information about real-world objects and environments with remote collaborators. While these calls provide valuable visual context in real time, the experience of interacting with people and moving around a space is significantly reduced when compared to co-located conversations. Recent work has demonstrated the potential of Mobile Augmented Reality applications to enable more spatial forms of collaboration across distance. To better understand the dynamics of mobile AR collaboration and how this medium compares against the status quo, we conducted a comparative structured observation study to analyze people’s perception of space and interaction with remote collaborators across mobile video calls and AR-based calls. Fourteen pairs of participants completed a spatial collaboration task using each medium. Through a mixed-methods analysis of session videos, transcripts, motion logs, post-task exercises, and interviews, we highlight how the choice of medium influences the roles and responsibilities that collaborators take on and the construction of a shared language for coordination. We discuss the importance of spatial reasoning with one’s body, how video calls help participants “be on the same page” more directly, and how AR calls enable both onsite and remote collaborators to engage with the space and each other in ways that resemble in-person interaction. Our study offers a nuanced view of the benefits and limitations of both mediums, and we conclude with a discussion of design implications for future systems that integrate mobile video and AR to better support spatial collaboration in its many forms.

The integration of eye-tracking into consumer-grade virtual reality (VR) headsets presents a transformative opportunity for assessing user mental states within simulated, immersive environments. However, the validity of this built-in technology must be established against gold-standard real-world eye-tracking systems. This study employs a novel paradigm using a physically moving object to evaluate the accuracy of dynamic smooth pursuit, a key oculomotor function in mental state assessment. We rigorously validated the performance of the HTC Vive Pro Eye’s integrated eye-tracker against the Tobii Pro Glasses 3 using a high-precision OptiTrack motion capture system as ground-truth for object position. Eight participants completed both 2D and 3D gaze-tracking tasks. In the 2D condition, they tracked a dot on a screen, while in the 3D condition, they tracked a physically moving object. The real-world object trajectories captured by OptiTrack were replicated within a VR environment. Gaze data from both the VR headset and the Tobii glasses were recorded simultaneously and compared to the OptiTrack baseline using Dynamic Time Warping (DTW) to quantify accuracy. Results revealed a task-dependent performance. In the 2D task, the Tobii glasses demonstrated significantly lower DTW distances, indicating superior accuracy. Conversely, in the 3D task, the VR headset significantly outperformed the glasses, showing a closer match to the real object trajectory. This suggests that while traditional eye-trackers excel in constrained 2D contexts, integrated VR eye-tracking is more accurate for naturalistic 3D gaze pursuit. We conclude that VR headset eye-tracking is not only a reliable but also a cost-effective tool for research, particularly offering enhanced performance for studies conducted within immersive 3D simulations.

During musical collaboration, visual cues are essential for communication between musicians. Extended Reality (XR) applications, often used with head-mounted displays like Augmented Reality (AR) glasses, can limit the field of view (FOV) of players. We conducted a study to investigate the effects of limited FOV on co-presence, gesture recognition, overall enjoyment, and reaction time. Initially, we observed experienced musicians collaborating informally with and without visual occlusion, noting that collaboration suffered with limited FOV. We then conducted a within-subjects study with 19 participants, comparing an unrestricted FOV holographic setup called HoloJam to Nreal AR glasses with a 52$^{\circ}$ limited FOV. In the AR setup, we tested two conditions: standard AR with a 52$^{\circ}$ FOV and a modified AR notification system called Mini Musicians. Results showed that HoloJam provided higher co-presence, quicker gesture recognition, and greater enjoyment. The Mini Musicians application reduced reaction time and maintained enjoyment compared to the standard AR setup. We conclude that limited FOV impacts musical collaboration, but notifications can improve reaction time and should be considered in future XR music collaborations.

We investigate the role of doppelgängers on foreign language anxiety, trustworthiness, and Spanish vocabulary recall. Participants (n = 31) completed three Spanish language lessons presented by a doppelgänger, a generic VH, or a disembodied voice in immersive or desktop VR; completed the Foreign Language Anxiety Classroom Scale (FLACS), Ohanian inventory on trustworthiness; and were assessed on vocabulary recall. Our findings suggest potential avenues for leverage with the design of VHs for foreign language learning.

Earlier research establishes the predominance of virtual over real textures, flavors, and scents when two different stimuli are presented to users. However, the effects of virtual priming, rather than simultaneous exposure, has not been thoroughly investigated. Experiential priming involves the presentation of one set of stimuli prior to a second set of stimuli. We examined whether experiential priming conducted in a virtual environment influences real-world flavor profile perception. The results of our preliminary work suggest that VR experiential priming produces perceptual differences in the real world and warrants further investigation.

To investigate how field of view (FOV) affects how players notice the communicative gestures of their partner's avatar in a musical task, we conducted an experiment that compares several AR technologies with varying FOV. We measured response time to communicative gestures, co-presence score, and task enjoyment in three different AR scenarios: holograms, AR headset, and an AR headset with notifications of the avatar's intention to gesture. Results suggest that the hologram setup had the fastest response time and highest ratings for sense of co-presence and task enjoyment. Notification tasks slowed response time, but noticeably improved co-presence with the avatar.