When Action Speaks Louder than Words: Exploring Non-Verbal and Paraverbal Features in Dyadic Collaborative VR
Open Access
Abstract
Soft skills such as communication and collaboration are vital in both professional and educational settings, yet difficult to train and assess objectively. Traditional role-playing scenarios rely heavily on subjective trainer evaluations—either in real time, where subtle behaviors are missed, or through time-intensive post hoc analysis. Virtual reality (VR) offers a scalable alternative by immersing trainees in controlled, interactive scenarios while simultaneously capturing fine-grained behavioral signals. This study investigates how task design in VR shapes non-verbal and paraverbal behaviors during dyadic collaboration. We compared two puzzle tasks: Task 1, which provided shared visual access and dynamic gesturing, and Task 2, which required verbal coordination through separation and turn-taking. From multimodal tracking data, we extracted features including gaze behaviors (eye contact, joint attention), hand gestures, facial expressions, and speech activity, and compared them across tasks. A clustering analysis explored whether o not tasks could be differentiated by their behavioral profiles. Results showed that Task 2, the more constrained condition, led participants to focus more visually on their own workspaces, suggesting that interaction difficulty can reduce partner-directed attention. Gestures were more frequent in shared-visual tasks, while speech became longer and more structured when turn-taking was enforced. Joint attention increased when participants relied on verbal descriptions rather than on a visible shared reference. These findings highlight how VR can elicit distinct soft skill behaviors through scenario design, enabling data-driven analysis of collaboration. This work contributes to scalable assessment frameworks with applications in training, adaptive agents, and human-AI collaboration.Keywords
Funding Information
- Flemish Government
This publication has 54 references indexed in Scilit:
- Developing conversational Virtual Humans for social emotion elicitation based on large language modelsExpert Systems with Applications, 2024
- The influence of perceptual load on gaze-induced attentional orienting: The modulation of expectationConsciousness and Cognition, 2023
- Corporeal Interactions inVRChat: Situational Intensity and Body SynchronizationSymbolic Interaction, 2023
- Revealing the theoretical basis of gamification: A systematic review and analysis of theory in research on gamification, serious games and game-based learningComputers in Human Behavior, 2021
- Virtual reality as a communication medium: a comparative study of forced compliance in virtual reality versus physical worldVirtual Reality, 2021
- Automatic multimodal assessment of soft skills in social interactions: a reviewMultimedia Tools and Applications, 2020
- Assessment and Teaching of 21st Century Skills2018
- Stress and anxiety detection using facial cues from videosBiomedical Signal Processing and Control, 2017
- The role of shared visual information for joint action coordinationCognition, 2016
- Social signal processing: Survey of an emerging domainImage and Vision Computing, 2009