Visual and linguistic information in gesture classification
Authors:
Open Access
- 13 October 2004
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 113-120
Abstract
Classification of natural hand gestures is usually approached by applying pattern recognition to the movements of the hand. However, the gesture categories most frequently cited in the psychology literature are fundamentally multimodal; the definitions make reference to the surrounding linguistic context. We address the question of whether gestures are naturally multimodal, or whether they can be classified from hand-movement data alone. First, we describe an empirical study showing that the removal of auditory information significantly impairs the ability of human raters to classify gestures. Then we present an automatic gesture classification system based solely on an n-gram model of linguistic context; the system is intended to supplement a visual classifier, but achieves 66% accuracy on a three-class classification problem on its own. This represents higher accuracy than human raters achieve when presented with the same information.Keywords
This publication has 10 references indexed in Scilit:
- Converting text into agent animations2004
- Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality2003
- Hand motion gestural oscillations and multimodal discourse2003
- Prosody based co-analysis for continuous recognition of coverbal gestures2003
- Catchments, prosody and discourseGesture, 2001
- Toward Natural Gesture/Speech Control of a Large Display2001
- BEAT2001
- Mutual disambiguation of recognition errors in a multimodel architecture1999
- A Framework for Gesture Generation and Interpretation1998
- QuickSet1997