Development of the Fully Automatic Robotic Storyteller – Comparing Manual, Semi-Automatic, and LLM-Based Emotional Story Annotation for Multimodal Robotic Storytelling
- 11 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. PP (3) , 1-18
Abstract
Storytelling has accompanied mankind since its beginnings and receiving stories is still a popular leisure activity. Today, new storytelling media are emerging. For example, social robots can act as storytellers, using different modalities such as facial expressions or body postures. With the goal of automating robotic storytelling, we annotated stories with emotion labels which a robot can use to automatically adapt its behavior to. Doing so, three different approaches and a baseline are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), 3) a LLM approach using ChatGPT (GPT), and 4) random annotations as baseline (RBL). In an online study showing videos of a storytelling robot, the annotations were validated using a backwards annotation task, with GPT closely followed by MA achieving the best, and the RBL the worst results. In a subsequent lab user study, all versions of the story were compared regarding storytelling experience, emotions, and robot perception, revealing no significant differences for any of these variables based on the annotation approach, but a significant decrease of negative affect and fatigue over the storytelling in all conditions. Taken together, the results indicate suitability of both the manual and GPT approaches. On this empirical basis, the Automatic Robotic Storyteller was implemented using GPT-based annotations and bodily expressions. The reduced programming effort of our system makes robotic storytelling accessible to a broader audience, strengthening its deployment in the field of affective computing.Keywords
This publication has 103 references indexed in Scilit:
- The Chronicles of ChatGPT: Generating and Evaluating Visual Novel Narratives on Climate Change Through ChatGPT2023
- CLARIN-Emo: Training Emotion Recognition Models Using Human Annotation and ChatGPT2023
- Introducing the Social Robot EBO: An Interactive and Socially Aware Storyteller Robot for Therapies with Older Adults2022
- Role of cognitive absorption in building user trust and experiencePsychology & Marketing, 2021
- Analysis of Influencing Factors on Humanoid Robots’ Emotion Expressions by Body Language2018
- Natural Human‐Robot Interaction2017
- There Once Was a Robot Storyteller: Measuring the Effects of Emotion and Non-verbal Behaviour2017
- Social Robotics2016
- “This story is right on”: The impact of regulatory fit on narrative engagement and persuasionEuropean Journal of Social Psychology, 2008
- Basic Emotions1999