Mind the Mix: Exploring the Cognitive Underpinnings of Multimodal Interaction in Augmented Reality Systems
Open Access
- 2 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Abstract
Exploring the intricate dynamics of Multimodal Interaction (MMI) in Augmented Reality (AR), this study presents a novel conceptual framework, crafted from a review of cognitive theories. Our research delves into how input modalities, output modalities, and their combinations uniquely influence user experiences in AR environments. Recognizing a gap in the existing MMI literature, especially within the AR context, we propose a conceptual framework to understand these complex relationships. Our framework pinpoints three critical factors: the choice of input modality, the verbal processing code of outputs, and the synergistic effects of input-output combinations. These elements are hypothesized to significantly impact user interaction and performance in AR systems. This work-in-progress not only contributes to the theoretical discourse in HCI but also sets the stage for future empirical investigations, aiming to enhance user-centered design in the evolving field of AR technology.Keywords
This publication has 29 references indexed in Scilit:
- EXTENDED REALITY (XR) ENVIRONMENTS2021
- INFORMATION PROCESSING2021
- Emerging features of modality mappings in task switching: modality compatibility requires variability at the level of both stimulus and response modalityPsychological Research, 2017
- The role of sensory-motor modality compatibility in language processingPsychological Research, 2015
- A usability study of multimodal input in an augmented reality environmentVirtual Reality, 2013
- Multiple Resource Theory2012
- Contemporary cognitive load theory research: The good, the bad and the uglyComputers in Human Behavior, 2011
- Multimodal interaction: A suitable strategy for including older users?Interacting with Computers, 2010
- An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures2006
- Working Memory1974