MULTIMODAL COMMUNICATION FROM MULTIMODAL THINKING — TOWARDS AN INTEGRATED MODEL OF SPEECH AND GESTURE PRODUCTION
Authors:
- 1 March 2008
- journal article
- research article
- Published by World Scientific Pub Co Pte Ltd in International Journal of Semantic Computing
- Vol. 02 (01) , 115-136
Abstract
A computational model for the automatic production of combined speech and iconic gesture is presented. The generation of multimodal behavior is grounded in processes of multimodal thinking, in which a propositional representation interacts and interfaces with an imagistic representation of visuo-spatial imagery. An integrated architecture for this is described, in which the planning of content and the planning of form across both modalities proceed in an interactive manner. Results from an empirical study are reported that inform the on-the-spot formation of gestures.Keywords
This publication has 26 references indexed in Scilit:
- Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions2007
- I see it in my hands’ eye: Representational gestures reflect conceptual demandsLanguage and Cognitive Processes, 2007
- Synthesizing multimodal utterances for conversational agentsComputer Animation and Virtual Worlds, 2004
- Microplanning with Communicative Intentions: The SPUD SystemComputational Intelligence, 2003
- How representational gestures help speaking2000
- Gesture, Speech, and Lexical Access: The Role of Lexical Movements in Speech ProductionPsychological Science, 1996
- Do Gestures Communicate? A ReviewResearch on Language and Social Interaction, 1994
- Word familiarity predicts temporal asynchrony of hand gestures and speech.Journal of Experimental Psychology: Learning, Memory, and Cognition, 1992
- Seeing and imagining in the cerebral hemispheres: A computational approach.Psychological Review, 1987
- Beyond pictures and words: Alternative information-processing models for imagery effect in verbal memory.Psychological Bulletin, 1978