GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
Authors:
- Xiyun HuXHXiyun Hu
- Dizhi MaDMDizhi Ma
- Fengming HeFHFengming He
- Zhengzhe ZhuZZZhengzhe Zhu
- Shao-Kang HsiaSHShao-Kang Hsia
- Chenfei ZhuCZChenfei Zhu
- Ziyi LiuZLZiyi Liu
- Karthik RamaniKRKarthik Ramani
Preprint
Abstract
Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation2024
- Conceptualising touch in VRVirtual Reality, 2021
- Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture RepresentationIEEE Access, 2020
- How What We See and What We Know Influence Iconic Gesture ProductionJournal of Nonverbal Behavior, 2017
- Gesture and speech in interaction: An overviewSpeech Communication, 2014
- A usability study of multimodal input in an augmented reality environmentVirtual Reality, 2013
- Deictic gestures and symbolic gestures produced by adults in an experimental context: Hand shapes and hand preferencesLaterality, 2013
- The differentiation of iconic and metaphoric gestures: Common and unique integration processesHuman Brain Mapping, 2011
- Augmented assembly technologies based on 3D bare-hand interactionCIRP Annals, 2011
- What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speakingJournal of Memory and Language, 2003