ASAP: Endowing Adaptation Capability to Agent in Human-Agent Interaction
Open Access
- 27 March 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 464-475
Abstract
Socially Interactive Agents (SIAs) offer users with interactive face-to-face conversations. They can take the role of a speaker and communicate verbally and nonverbally their intentions and emotional states; but they should also act as active listener and be an interactive partner. In human-human interaction, interlocutors adapt their behaviors reciprocally and dynamically. The endowment of such adaptation capability can allow SIAs to show social and engaging behaviors. In this paper, we focus on modelizing the reciprocal adaptation to generate SIA behaviors for both conversational roles of speaker and listener. We propose the Augmented Self-Attention Pruning (ASAP) neural network model. ASAP incorporates recurrent neural network, attention mechanism of transformers, and pruning technique to learn the reciprocal adaptation via multimodal social signals. We evaluate our work objectively, via several metrics, and subjectively, through a user perception study where the SIA behaviors generated by ASAP is compared with those of other state-of-the-art models. Our results demonstrate that ASAP significantly outperforms the state-of-the-art models and thus shows the importance of reciprocal adaptation modeling.Keywords
Funding Information
- IA ANR-DFG-JST Panorama (ANR-20-IADJ-0008)
- Sorbonne Université
- ANR-JST-CREST TAPAS (ANR-19-JSTS-0001)
This publication has 46 references indexed in Scilit:
- OpenFace: An open source facial behavior analysis toolkit2016
- Predicting Short-Term Traffic Flow by Long Short-Term Memory Recurrent Neural Network2015
- The appearance effect: Influences of virtual agent features on performance and motivationComputers in Human Behavior, 2015
- Head motion synthesis from speech using deep neural networksMultimedia Tools and Applications, 2014
- You Give Me the ChillsPsychological Science, 2012
- A probabilistic multimodal approach for predicting listener backchannelsAutonomous Agents and Multi-Agent Systems, 2009
- A Spoken Dialog System for Chat-Like Conversations Considering Response Timing2007
- Dynamic Time Warping2007
- Framewise phoneme classification with bidirectional LSTM and other neural network architecturesNeural Networks, 2005
- The Kolmogorov-Smirnov Test for Goodness of FitJournal of the American Statistical Association, 1951