Analyzing Multifunctionality of Head Movements in Face-to-Face Conversations Using Deep Convolutional Neural Networks
Open Access
- 1 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 217169-217195
Abstract
A functional head-movement corpus and convolutional neural networks (CNNs) for detecting head-movement functions are presented for analyzing the multiple communicative functions of head movements in multiparty face-to-face conversations. First, focusing on the multifunctionality of head movements, i.e., that a single head movement can simultaneously perform multiple functions, this paper defines 32 non-mutually-exclusive function categories, whose genres are speech production, eliciting and giving feedback, turn management, and cognitive and affect display. To represent and capture arbitrary multifunctional structures, our corpus employs multiple binary codes and logical-sum-based aggregations of multiple coders’ judgments. A corpus analysis targeting four-party Japanese conversations revealed multifunctional patterns in which the speaker modulates multiple functions, such as emphasis and eliciting listeners’ responses, through rhythmic head movements, and listeners express various attitudes and responses through continuous back-channel head movements. This paper proposes CNN-based binary classifiers for detecting each of the functions from the angular velocity of the head pose and the presence or absence of utterances. The experimental results showed that the recognition performance varies greatly, from approximately 30% to 90% in terms of the F-score, depending on the function category, and the performance was positively correlated with the amount of data and inter-coder agreement. In addition, we noted a tendency toward overdetection that added more functions to those originally in the corpus. The analyses and experiments confirm that our approach is promising for studying the multifunctionality of head movements.Keywords
Funding Information
- NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, under a joint research agreement between NTT
- Yokohama National University
This publication has 113 references indexed in Scilit:
- Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: A survey of related cues, databases, and toolsImage and Vision Computing, 2013
- Analyzing nonverbal listener responses using parallel recordings of multiple listenersCognitive Processing, 2012
- Robust classification of face and head gestures in videoImage and Vision Computing, 2011
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- Social signal processing: Survey of an emerging domainImage and Vision Computing, 2009
- Automatic nonverbal analysis of social interaction in small groups: A reviewImage and Vision Computing, 2009
- Phonetics and social action in agreements and disagreementsJournal of Pragmatics, 2006
- Listener Responses as a Collaborative Process: The Role of GazeJournal of Communication, 2002
- Listeners as co-narrators.Journal of Personality and Social Psychology, 2000
- On back-channel behavior in Japanese and English casual conversationLinguistics, 1986