What's in a gaze?
Authors:
Open Access
- 13 January 2008
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 20-29
Abstract
Multimodal conversational interfaces allow users to carry a dialog with a graphical display using speech to accomplish a particular task. Motivated by previous psycholinguistic findings, we examine how eye-gaze contributes to reference resolution in such a setting. Specifically, we present an integrated probabilistic framework that combines speech and eye-gaze for reference resolution. We further examine the relationship between eye-gaze and increased domain modeling with corresponding linguistic processing. Our empirical results show that the incorporation of eye-gaze significantly improves reference resolution performance. This improvement is most dramatic when a simple domain model is used. Our results also show that minimal domain modeling combined with eye-gaze significantly outperforms complex domain modeling without eye-gaze, which indicates that eye-gaze can be used to potentially compensate a lack of domain modeling for reference resolution.Keywords
Funding Information
- Division of Information and Intelligent Systems (IIS-0535112IIS-0347548)
This publication has 14 references indexed in Scilit:
- Conversing with the user based on eye-gaze patterns2005
- Utilizing Visual Attention for Cross-Modal Coreference InterpretationLecture Notes in Computer Science, 2005
- RealTourist – A Study of Augmenting Human-Human and Human-Computer Dialogue with Eye-Gaze Overlay2005
- Visual Salience and Reference Resolution in Simulated 3-D EnvironmentsArtificial Intelligence Review, 2004
- Optimization in multimodal interpretation2004
- Resolving pronominal reference to abstract entities2001
- Finite-state multimodal parsing and understanding2000
- Integration of Visual and Linguistic Information in Spoken Language ComprehensionScience, 1995
- Eye fixations and cognitive processesCognitive Psychology, 1976
- Saccadic suppression: A review and an analysis.Psychological Bulletin, 1974