Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations
Authors:
- 12 March 2023
- journal article
- research article
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 34 (6)
Abstract
The present work investigates the effect of natural conversations with virtual beings on user perceptions with a current conversational AI model (Meta's BlenderBot). To this aim, we designed a virtual being from a deep learning‐generated face and a conversational AI model acting as a virtual conversation partner in an online conferencing software and evaluated it in 11 perceptions of social attributes. Compared to prior expectations, participants perceived the virtual being as distinctly higher in warmth (engaging, empathic, and approachable) but lower in realism and credibility after 5 days of 10 min daily conversations (Study 1). Further, we explored the idea of simplifying the technical setup to reduce the technical entry barrier for such AI applications (Study 2). To this aim, we conducted several trials of fine‐tuning a small conversational model of 90 million parameters until its performance metrics improved. Testing this fine‐tuned model with users revealed that this model was not perceived differently from a large conversational model (1.4 billion parameters). In summary, our findings show that recent progress in conversational AI has added warmth‐related aspects to the user experience with virtual beings, and that fine‐tuning a conversational AI model can be effective to reduce technical complexity.Keywords
Funding Information
- Yonsei University (2021‐22‐0317)
This publication has 96 references indexed in Scilit:
- Perceptual and Category Processing of the Uncanny Valley Hypothesis' Dimension of Human Likeness: Some Methodological IssuesJournal of Visualized Experiments, 2013
- Does the uncanny valley exist? An empirical test of the relationship between eeriness and the human likeness of digitally created facesComputers in Human Behavior, 2013
- Virtual agents in retail web sites: Benefits of simulated social interaction for older usersComputers in Human Behavior, 2012
- Socially Anxious and Confident Men Interact with a Forward Virtual Woman: An Experimental StudyPLOS ONE, 2012
- Facial expression of emotion and perception of the Uncanny Valley in virtual charactersComputers in Human Behavior, 2011
- The Uncanny Valley: Effect of Realism on the Impression of Artificial Human FacesPRESENCE: Virtual and Augmented Reality, 2007
- RE: "A DICTIONARY OF EPIDEMIOLOGY, FOURTH EDITION, EDITED BY JOHN M. LAST, ROBERT A. SPASOFF, AND SUSAN G. HARRIS"American Journal of Epidemiology, 2001
- On the Dominance of Moral Categories in Impression FormationPersonality and Social Psychology Bulletin, 1998
- Fitting Continuous Models to Effect Size DataJournal of Educational Statistics, 1982
- Coefficient Alpha and the Internal Structure of TestsPsychometrika, 1951