Speaker Intimacy Estimation in Chat-Talks Based on Verbal and Non-Verbal Information
Authors:
Open Access
- 27 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. PP, 1
Abstract
Conversations based on mutual intimacy are critical for maintaining positive relationships. A detailed understanding of speaker relationships in dialogues enhances various applications, such as information recommendation systems. Such systems, when interacting with multiple users, can provide more tailored information by understanding the users’ relationships. Furthermore, dialogue systems, which are becoming increasingly prevalent in society, can foster long-term user engagement by recognizing and responding to the intimacy levels of users. This study explores a method for estimating the intimacy levels of speakers and dialogue partners in conversational exchanges. Our approach utilizes a multimodal corpus of natural conversations with 71 Japanese participants, complete with metadata indicating each speaker’s perceived intimacy level. We identified key features for estimating intimacy by analyzing the statistical parameters of these features. Our comprehensive analysis encompassed both verbal and non-verbal information, including prosody, gestures, and facial expressions. The proposed intimacy estimation model combines multimodal features using a multi-stream Bi-directional Long Short-Term Memory (BLSTM) network and grasps the contextual information of conversations with a Context BLSTM. Our model’s effectiveness is demonstrated through comparisons with several baseline models. Experimental results show that our proposed model significantly improves the overall performance compared with other models. Although the RoBERTa-based method (the best baseline model) achieved an F1 score of 0.571, our method had an F1 score of 0.594. In particular, an ablation study shows that combining verbal and non-verbal features is useful for intimacy estimation. The performance was further improved by extending the dialogue context, showing that the proposed model can estimate three levels of intimacy with an F1 score of 0.666 by observing eight utterance exchanges.Keywords
Funding Information
- Japan Society for the Promotion of Science (JP20K19903)
This publication has 46 references indexed in Scilit:
- Interpersonal relation recognition: a surveyMultimedia Tools and Applications, 2022
- SRE-Net Model for Automatic Social Relation Extraction from Video2018
- From Facial Expression Recognition to Interpersonal Relation PredictionInternational Journal of Computer Vision, 2017
- An estimation model for social relationship strength based on users' profiles, co-occurrence and interaction activitiesNeurocomputing, 2016
- Reflections on Historical Trends and Prospects in Contemporary Nonverbal ResearchJournal of Nonverbal Behavior, 2014
- Am I acceptable to you? Effect of a robot’s verbal language forms on people’s social distance from robotsComputers in Human Behavior, 2013
- Estimation of friendship and hierarchy from conversation recordsInformation Sciences, 2009
- Negotiated Collusion: Modeling Social Language and its Relationship Effects in Intelligent AgentsUser Modeling and User-Adapted Interaction, 2003
- The Importance of Nonverbal Cues in Judging RapportJournal of Nonverbal Behavior, 1999
- A sociolinguistic analysis of the Japanese honorificsJournal of Pragmatics, 1986