Investigating the Multimodal Nature of Human Communication
Authors:
- 1 January 2009
- journal article
- Published by Hogrefe Publishing Group in Journal of Psychophysiology
- Vol. 23 (2) , 63-76
Abstract
The multimodal nature of human communication has been well established. Yet few empirical studies have systematically examined the widely held belief that this form of perception is facilitated in comparison to unimodal or bimodal perception. In the current experiment we first explored the processing of unimodally presented facial expressions. Furthermore, auditory (prosodic and/or lexical-semantic) information was presented together with the visual information to investigate the processing of bimodal (facial and prosodic cues) and multimodal (facial, lexic, and prosodic cues) human communication. Participants engaged in an identity identification task, while event-related potentials (ERPs) were being recorded to examine early processing mechanisms as reflected in the P200 and N300 component. While the former component has repeatedly been linked to physical property stimulus processing, the latter has been linked to more evaluative “meaning-related” processing. A direct relationship between P200 and N300 amplitude and the number of information channels present was found. The multimodal-channel condition elicited the smallest amplitude in the P200 and N300 components, followed by an increased amplitude in each component for the bimodal-channel condition. The largest amplitude was observed for the unimodal condition. These data suggest that multimodal information induces clear facilitation in comparison to unimodal or bimodal information. The advantage of multimodal perception as reflected in the P200 and N300 components may thus reflect one of the mechanisms allowing for fast and accurate information processing in human communication.Keywords
This publication has 52 references indexed in Scilit:
- Time course and specificity of event-related potentials to emotional expressionsNeuroReport, 2004
- Recognition of affective prosody: Continuous wavelet measures of event‐related brain potentials to emotional exclamationsPsychophysiology, 2003
- Early processing of the six basic facial emotional expressionsCognitive Brain Research, 2003
- Face-selective processing and the effect of pleasant and unpleasant emotional expressions on ERP correlatesInternational Journal of Psychophysiology, 2003
- Discrimination of emotional facial expressions in a visual oddball task: an ERP studyBiological Psychology, 2002
- Neural systems for recognizing emotionCurrent Opinion in Neurobiology, 2002
- Lesions of the human amygdala impair enhanced perception of emotionally salient eventsNature, 2001
- Relationships among Facial, Prosodic, and Lexical Channels of Emotional Perceptual ProcessingCognition and Emotion, 2000
- Crossmodal identificationTrends in Cognitive Sciences, 1998
- American Electroencephalographic Society Guidelines for Standard Electrode Position NomenclatureJournal Of Clinical Neurophysiology, 1991