Immersive machine learning for social attitude detection in virtual reality narrative games
Open Access
- 7 April 2022
- journal article
- research article
- Published by Springer Nature in Virtual Reality
- Vol. 26 (4) , 1519-1538
Abstract
People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.Keywords
Funding Information
- Innovate UK (TS/S02221X/1)
- Engineering and Physical Sciences Research Council (EP/L015846/1)
This publication has 38 references indexed in Scilit:
- Toward a second-person neuroscienceBehavioral and Brain Sciences, 2013
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Narrative Structures in Computer and Video Games: Part 1: Context, Definitions, and Initial FindingsGames and Culture, 2010
- Place illusion and plausibility can lead to realistic behaviour in immersive virtual environmentsPhilosophical Transactions of the Royal Society B: Biological Sciences, 2009
- Affect-aware tutors: recognising and responding to student affectInternational Journal of Learning Technology, 2009
- Eyes aliveACM Transactions on Graphics, 2002
- A Virtual Presence CounterPRESENCE: Virtual and Augmented Reality, 2000
- Implicit human computer interaction through contextPersonal and Ubiquitous Computing, 2000
- Long Short-Term MemoryNeural Computation, 1997
- Adaptation in Dyadic Interaction: Defining and Operationalizing Patterns of Reciprocity and CompensationCommunication Theory, 1993