Sophia C. Steinhaeusser
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Sophia C. Steinhaeusser has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
- Birgit Lugrin
Socially Interactive Agents, University of Würzburg, Würzburg, Germany
12 shared publications Birgit Lugrin
Socially Interactive AgentsUniversity of Wrzburg
3 shared publications- Sophia Maier
Https://ror.org/00fbnyb24
3 shared publications - Albin Zehe
Center for Artificial Intelligence and Data Science (CAIDAS), Julius-Maximilians-Universität Würzburg
2 shared publications - Elisabeth Ganal
University of Würzburg
2 shared publications - Marc Erich Latoschik
Human-Computer Interaction Group, University of Wurzburg, Wurzburg, Germany
2 shared publications
Latest Publications
- Open Access
- 26 June 2026
- Published by Science Exploration Press in Empathic Computing
- Vol. 2 (3) , 202605
Effects of a robotic storytelling intervention integrating music and sound effects on prejudice toward mental illness
Aims: Mental illnesses affect millions worldwide. Despite greater public awareness, significant stigma remains. Anti-stigmatization interventions can help reduce prejudice by promoting education and meaningful contact with people with mental illness. Technology-based interventions may further support these efforts by simulating such contact through approaches such as storytelling. Social robots may offer additional advantages for storytelling-based interventions, especially through the integration of music and sound effects. Methods: In multimodal robotic storytelling, non-speech sounds remain largely overlooked, despite their importance in related media. To address this gap, we compared a robotic storyteller narrating a story about a person experiencing a panic attack using only voice and bodily expression with versions integrating sound effects, background music, or both. Results: While the addition of sounds did not affect recipients’ prejudice differently than storytelling without them, adding both sound effects and music increased transportation and improved associative empathy. In contrast, adding only music decreased associative empathy. Although mediation was not indicated, the results revealed transportation as a predictor for empathy and prejudice, while being only minimally influenced by sound integration itself. Conclusion: The effects of sound integration in a robotic storytelling intervention on recipients’ prejudice were mixed, recommending either the combination of both sound types or complete omission. Transportation was indicated as a key lever for increasing empathy and decreasing prejudice that warrants further investigation. Future work is needed to gain deeper insights into robotic storytelling as an intervention tool for reducing prejudice, including work on transportation as a modifiable factor and the integration of pre-post-measurements.
- 26 June 2026
- Published by Science Exploration Press in Empathic Computing
- Vol. 2 (3) , 202605
Open AccessEffects of a robotic storytelling intervention integrating music and sound effects on prejudice toward mental illness
Aims: Mental illnesses affect millions worldwide. Despite greater public awareness, significant stigma remains. Anti-stigmatization interventions can help reduce prejudice by promoting education and meaningful contact with people with mental illness. Technology-based interventions may further support these efforts by simulating such contact through approaches such as storytelling. Social robots may offer additional advantages for storytelling-based interventions, especially through the integration of music and sound effects. Methods: In multimodal robotic storytelling, non-speech sounds remain largely overlooked, despite their importance in related media. To address this gap, we compared a robotic storyteller narrating a story about a person experiencing a panic attack using only voice and bodily expression with versions integrating sound effects, background music, or both. Results: While the addition of sounds did not affect recipients’ prejudice differently than storytelling without them, adding both sound effects and music increased transportation and improved associative empathy. In contrast, adding only music decreased associative empathy. Although mediation was not indicated, the results revealed transportation as a predictor for empathy and prejudice, while being only minimally influenced by sound integration itself. Conclusion: The effects of sound integration in a robotic storytelling intervention on recipients’ prejudice were mixed, recommending either the combination of both sound types or complete omission. Transportation was indicated as a key lever for increasing empathy and decreasing prejudice that warrants further investigation. Future work is needed to gain deeper insights into robotic storytelling as an intervention tool for reducing prejudice, including work on transportation as a modifiable factor and the integration of pre-post-measurements.
- 11 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. PP (3) , 1-18
Development of the Fully Automatic Robotic Storyteller – Comparing Manual, Semi-Automatic, and LLM-Based Emotional Story Annotation for Multimodal Robotic Storytelling
Storytelling has accompanied mankind since its beginnings and receiving stories is still a popular leisure activity. Today, new storytelling media are emerging. For example, social robots can act as storytellers, using different modalities such as facial expressions or body postures. With the goal of automating robotic storytelling, we annotated stories with emotion labels which a robot can use to automatically adapt its behavior to. Doing so, three different approaches and a baseline are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), 3) a LLM approach using ChatGPT (GPT), and 4) random annotations as baseline (RBL). In an online study showing videos of a storytelling robot, the annotations were validated using a backwards annotation task, with GPT closely followed by MA achieving the best, and the RBL the worst results. In a subsequent lab user study, all versions of the story were compared regarding storytelling experience, emotions, and robot perception, revealing no significant differences for any of these variables based on the annotation approach, but a significant decrease of negative affect and fatigue over the storytelling in all conditions. Taken together, the results indicate suitability of both the manual and GPT approaches. On this empirical basis, the Automatic Robotic Storyteller was implemented using GPT-based annotations and bodily expressions. The reduced programming effort of our system makes robotic storytelling accessible to a broader audience, strengthening its deployment in the field of affective computing.
- 11 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. PP (3) , 1-18
Development of the Fully Automatic Robotic Storyteller – Comparing Manual, Semi-Automatic, and LLM-Based Emotional Story Annotation for Multimodal Robotic Storytelling
Storytelling has accompanied mankind since its beginnings and receiving stories is still a popular leisure activity. Today, new storytelling media are emerging. For example, social robots can act as storytellers, using different modalities such as facial expressions or body postures. With the goal of automating robotic storytelling, we annotated stories with emotion labels which a robot can use to automatically adapt its behavior to. Doing so, three different approaches and a baseline are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), 3) a LLM approach using ChatGPT (GPT), and 4) random annotations as baseline (RBL). In an online study showing videos of a storytelling robot, the annotations were validated using a backwards annotation task, with GPT closely followed by MA achieving the best, and the RBL the worst results. In a subsequent lab user study, all versions of the story were compared regarding storytelling experience, emotions, and robot perception, revealing no significant differences for any of these variables based on the annotation approach, but a significant decrease of negative affect and fatigue over the storytelling in all conditions. Taken together, the results indicate suitability of both the manual and GPT approaches. On this empirical basis, the Automatic Robotic Storyteller was implemented using GPT-based annotations and bodily expressions. The reduced programming effort of our system makes robotic storytelling accessible to a broader audience, strengthening its deployment in the field of affective computing.
- 2 January 2026
- Published by Springer Nature
- p. 174-193
Message for You: Observing the Effects of a Social Robot’s Interruptions During an Office Task
- 2 January 2026
- Published by Springer Nature
- p. 174-193
Message for You: Observing the Effects of a Social Robot’s Interruptions During an Office Task
- Open Access
- 16 December 2025
- Published by Elsevier in Computers in Human Behavior: Artificial Humans
- Vol. 7
Multimodal robotic storytelling integrating sound effects and background music
- 16 December 2025
- Published by Elsevier in Computers in Human Behavior: Artificial Humans
- Vol. 7
Open AccessMultimodal robotic storytelling integrating sound effects and background music
- 29 November 2025
- Published by Springer Nature
- p. 72-94
Social Robots Against Bullying – Effects of Embodiment and Interactivity on Social Story Experience and Efficiency
- 29 November 2025
- Published by Springer Nature
- p. 72-94
Social Robots Against Bullying – Effects of Embodiment and Interactivity on Social Story Experience and Efficiency
- Open Access
- 1 January 2025
- Published by Science Exploration Press in Empathic Computing
- Vol. 1 (2) , 202404
Integrating colored lights into multimodal robotic storytelling
Aims: Storytelling has evolved alongside human culture, giving rise to new media such as social robots. While these robots employ modalities similar to those used by humans, they can also utilize non-biomimetic modalities, such as color, which are commonly associated with emotions. As research on the use of colored light in robotic storytelling remains limited, this study investigates its integration through three empirical studies. Methods: We conducted three studies to explore the impact of colored light in robotic storytelling. The first study examined the effect of emotion-inducing colored lighting in romantic storytelling. The second study employed an online survey to determine appropriate light colors for specific emotions, based on images of the robot’s emotional expressions. The third study compared four lighting conditions in storytelling: emotion-driven colored lights, context-based colored lights, constant white light, and no additional lighting. Results: The first study found that while colored lighting did not significantly influence storytelling experience or perception of the robot, it made recipients felt more serene. The second study showed improved recognition of amazement, rage, and neutral emotional states when colored light accompanied body language. The third study revealed no significant differences across lighting conditions in terms of storytelling experience, emotions, or robot perception; however, participants generally appreciated the use of colored lights. Emotion-driven lighting received slightly more favorable subjective evaluations. Conclusion: Colored lighting can enhance the emotional expressiveness of robots. Both emotion- driven and context-based lighting strategies are appropriate for robotic storytelling. Through this series of studies, we contribute to the understanding of how colored lights are perceived in robotic communication, particularly within storytelling contexts.
- 1 January 2025
- Published by Science Exploration Press in Empathic Computing
- Vol. 1 (2) , 202404
Open AccessIntegrating colored lights into multimodal robotic storytelling
Aims: Storytelling has evolved alongside human culture, giving rise to new media such as social robots. While these robots employ modalities similar to those used by humans, they can also utilize non-biomimetic modalities, such as color, which are commonly associated with emotions. As research on the use of colored light in robotic storytelling remains limited, this study investigates its integration through three empirical studies. Methods: We conducted three studies to explore the impact of colored light in robotic storytelling. The first study examined the effect of emotion-inducing colored lighting in romantic storytelling. The second study employed an online survey to determine appropriate light colors for specific emotions, based on images of the robot’s emotional expressions. The third study compared four lighting conditions in storytelling: emotion-driven colored lights, context-based colored lights, constant white light, and no additional lighting. Results: The first study found that while colored lighting did not significantly influence storytelling experience or perception of the robot, it made recipients felt more serene. The second study showed improved recognition of amazement, rage, and neutral emotional states when colored light accompanied body language. The third study revealed no significant differences across lighting conditions in terms of storytelling experience, emotions, or robot perception; however, participants generally appreciated the use of colored lights. Emotion-driven lighting received slightly more favorable subjective evaluations. Conclusion: Colored lighting can enhance the emotional expressiveness of robots. Both emotion- driven and context-based lighting strategies are appropriate for robotic storytelling. Through this series of studies, we contribute to the understanding of how colored lights are perceived in robotic communication, particularly within storytelling contexts.
- Open Access
- 19 November 2024
- Published by Association for Computing Machinery (ACM) in Games: Research and Practice
- Vol. 2 (4) , 1-26
Pushing Yourself to the Limit - Influence of Emotional Virtual Environment Design on Physical Training in VR
The design of virtual environments (VEs) can strongly influence users’ emotions. These VEs are also an important aspect of immersive Virtual Reality (VR) exergames – training system that can inspire athletes to train in a highly motivated way and achieve a higher training intensity. VR-based training and rehabilitation systems can increase a user’s motivation to train and to repeat physical exercises. The surrounding VE can potentially predominantly influence users’ motivation and hence potentially even physical performance. Besides providing potentially motivating environments, physical training can be enhanced by gamification. However, it is unclear whether the surrounding VE of a VR-based physical training system influences the effectiveness of gamification. We investigate whether an emotional positive or emotional negative design influences the sport performance and interacts with the positive effects of gamification. In a user study, we immerse participants in VEs following either an emotional positive, neutral, or negative design and measure the duration the participants can hold a static strength-endurance exercise. The study targeted the investigation of the effects of (1) emotional VE design as well as the (2) presence and absence of gamification. We did not observe significant differences in the performance of the participants independent of the conditions of VE design or gamification. Gamification caused a dominating effect on emotion and motivation over the emotional design of the VEs, thus indicating an overall positive impact. The emotional design influenced the participants’ intrinsic motivation but caused mixed results with respect to emotion. Overall, our results indicate the importance of using gamification and support the commonly used emotional positive VEs for physical training. They further indicate that the design space could also include other emotional directions of VE design.
- 19 November 2024
- Published by Association for Computing Machinery (ACM) in Games: Research and Practice
- Vol. 2 (4) , 1-26
Open AccessPushing Yourself to the Limit - Influence of Emotional Virtual Environment Design on Physical Training in VR
The design of virtual environments (VEs) can strongly influence users’ emotions. These VEs are also an important aspect of immersive Virtual Reality (VR) exergames – training system that can inspire athletes to train in a highly motivated way and achieve a higher training intensity. VR-based training and rehabilitation systems can increase a user’s motivation to train and to repeat physical exercises. The surrounding VE can potentially predominantly influence users’ motivation and hence potentially even physical performance. Besides providing potentially motivating environments, physical training can be enhanced by gamification. However, it is unclear whether the surrounding VE of a VR-based physical training system influences the effectiveness of gamification. We investigate whether an emotional positive or emotional negative design influences the sport performance and interacts with the positive effects of gamification. In a user study, we immerse participants in VEs following either an emotional positive, neutral, or negative design and measure the duration the participants can hold a static strength-endurance exercise. The study targeted the investigation of the effects of (1) emotional VE design as well as the (2) presence and absence of gamification. We did not observe significant differences in the performance of the participants independent of the conditions of VE design or gamification. Gamification caused a dominating effect on emotion and motivation over the emotional design of the VEs, thus indicating an overall positive impact. The emotional design influenced the participants’ intrinsic motivation but caused mixed results with respect to emotion. Overall, our results indicate the importance of using gamification and support the commonly used emotional positive VEs for physical training. They further indicate that the design space could also include other emotional directions of VE design.
- Open Access
- 30 July 2024
- Published by Elsevier in Computers in Human Behavior: Artificial Humans
- Vol. 2 (2)
Integrating sound effects and background music in Robotic storytelling – A series of online studies across different story genres
- 30 July 2024
- Published by Elsevier in Computers in Human Behavior: Artificial Humans
- Vol. 2 (2)
Open AccessIntegrating sound effects and background music in Robotic storytelling – A series of online studies across different story genres
- Open Access
- 4 April 2024
- Published by Springer Nature in Journal on Multimodal User Interfaces
- Vol. 18 (4) , 1-23
Towards the development of an automated robotic storyteller: comparing approaches for emotional story annotation for non-verbal expression via body language
Storytelling is a long-established tradition and listening to stories is still a popular leisure activity. Caused by technization, storytelling media expands, e.g., to social robots acting as multi-modal storytellers, using different multimodal behaviours such as facial expressions or body postures. With the overarching goal to automate robotic storytelling, we have been annotating stories with emotion labels which the robot can use to automatically adapt its behavior. With it, three different approaches are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), and 3) a machine learning based approach (ML). In an online study showing videos of a storytelling robot, the annotations were validated, with LIWC and MA achieving the best, and ML the worst results. In a laboratory user study, the three versions of the story were compared regarding transportation and cognitive absorption, revealing no significant differences but a positive trend towards MA. On this empirical basis, the Automated Robotic Storyteller was implemented using manual annotations. Future iterations should include other robots and modalities, fewer emotion labels and their probabilities.
- 4 April 2024
- Published by Springer Nature in Journal on Multimodal User Interfaces
- Vol. 18 (4) , 1-23
Open AccessTowards the development of an automated robotic storyteller: comparing approaches for emotional story annotation for non-verbal expression via body language
Storytelling is a long-established tradition and listening to stories is still a popular leisure activity. Caused by technization, storytelling media expands, e.g., to social robots acting as multi-modal storytellers, using different multimodal behaviours such as facial expressions or body postures. With the overarching goal to automate robotic storytelling, we have been annotating stories with emotion labels which the robot can use to automatically adapt its behavior. With it, three different approaches are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), and 3) a machine learning based approach (ML). In an online study showing videos of a storytelling robot, the annotations were validated, with LIWC and MA achieving the best, and ML the worst results. In a laboratory user study, the three versions of the story were compared regarding transportation and cognitive absorption, revealing no significant differences but a positive trend towards MA. On this empirical basis, the Automated Robotic Storyteller was implemented using manual annotations. Future iterations should include other robots and modalities, fewer emotion labels and their probabilities.