Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution
Preprint
Abstract
Understanding transcripts of immersive multimodal conversations is challenging because speakers frequently rely on visual context and non-verbal cues, such as gestures and visual attention, which are not captured in speech alone. This lack of information makes coreferences resolution-the task of linking ambiguous expressions like ``it'' or ``there'' to their intended referents-particularly challenging. In this paper we present a system that augments VR speech transcript with eye-tracking laser pointing data, and scene metadata to generate textual descriptions of non-verbal communication and the corresponding objects of interest. To evaluate the system, we collected gaze, gesture, and voice data from 12 participants (6 pairs) engaged in an open-ended design critique of a 3D model of an apartment. Our results show a 26.5\% improvement in coreference resolution accuracy by a GPT model when using our multimodal transcript compared to a speech-only baseline.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- VD-PCR: Improving visual dialog with pronoun coreference resolutionPattern Recognition, 2022
- The combination of visual communication cues in mixed reality remote collaborationJournal on Multimodal User Interfaces, 2020
- Evaluating the Impact of Virtual Reality on Design Review MeetingsJournal of Computing in Civil Engineering, 2020
- Look together: using gaze for assisting co-located collaborative searchPersonal and Ubiquitous Computing, 2016
- What Are You Talking About? Text-to-Image Coreference2014
- Support for deictic pointing in CVEs2014
- Real-time mutual gaze perception enhances collaborative learning and collaboration qualityInternational Journal of Computer-Supported Collaborative Learning, 2013
- Where are you pointing?2010
- A method to study visual attention aspects of collaboration2008
- Speech-augmented eye gaze interaction with small closely spaced targets2006