Modeling the Gesture-Speech Relation Through Novel Datasets for Multimodal Signal Analysis
Preprint
- 25 September 2025
- preprint
- Published by MDPI AG in Preprints.org
Abstract
We present preliminary results of a new methodology to study co-speech gesture in relation to specific linguistic structures. We draw on a large-scale video repository with time-aligned transcripts to build corpora in which the same linguistic expression is uttered by different speakers across multiple clips. We then extract dynamic coordinates of key body points to model their variation in relation with what is being said. In this paper, we present analyses of the distribution of wrist motion in gesture space for a dataset of 379 videos with utterances of 44 deictic time expressions in English (words or phrases pointing at the past, present, or future in relation to a center of temporal reference, e.g. “yesterday/today/tomorrow”). Even overall distributions of wrist positions in peripheral areas of gestural motion turn out to be influenced by these semantic distinctions. More fine-grained models are to be expected from the reconstruction of gestural trajectories, based on the chronological sequence of positions detected in each video. These initial results already suggest that so-called non-verbal behavior is deeply structured and attuned to language, quite beyond our current understanding. Once scaled up, such models have the potential to dramatically change any technologies connected to human communicative behavior.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressionsPLOS ONE, 2020
- Multimodal Language Processing in Human CommunicationTrends in Cognitive Sciences, 2019
- OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Toward an infrastructure for data-driven multimodal communication researchLinguistics Vanguard, 2018
- Utterance Construction Grammar (UCxG) and the variable multimodality of constructionsLinguistics Vanguard, 2017
- Multimodal constructs – multimodal constructions? The role of constructions in the working memoryLinguistics Vanguard, 2017
- The origin of human multi-modal communicationPhilosophical Transactions of the Royal Society B: Biological Sciences, 2014
- The processing of speech, gesture, and action during language comprehensionPsychonomic Bulletin & Review, 2014
- CQPweb — combining power, flexibility and usability in a corpus analysis toolCorpus Studies of Language Through Time, 2012