Coordinate-Aware Head–Hand Motion as Behavioral Process Data Across Immersive Interaction Contexts
Authors:
- Fabiana PeresFPFabiana Peres
- João Marcelo TeixeiraJTJoão Marcelo Teixeira
- Fátima L. S. NunesFNFátima L. S. Nunes
Preprint
- 28 August 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Immersive virtual and mixed reality systems continuously record head and hand/controller motion, yet these traces are commonly reduced to aggregate task outcomes or retained only for technical purposes. This study investigates whether coordinate-aware motion telemetry can instead serve as behavioral process data for understanding how interaction unfolds over time. We harmonized telemetry from four immersive contexts: full-scale VR assembly (FAST; 108 participants, 216 sessions), physically grounded MR/VR driving in a real vehicle (40 participants, 80 paired runs), educational cartography in GeoConnect (206 performance-linked stages from 35 contributing sessions), and virtual logic-circuit construction (177 sessions). The analysis combined kinematic, head-relative, bilateral, temporal-state, and performance-grounded descriptors while explicitly preserving the coordinate semantics of each task. FAST exhibited a robust bimanual-centered temporal profile, with 80.6\% of analyzed windows classified as bimanual. Repeated exposure was associated with shorter completion time and reduced head and left-hand movement, but increased right-hand specialization. In the driving context, world-space HMD paths exceeded 3~km despite participants remaining physically seated, demonstrating that mathematically valid trajectories can be behaviorally misleading when coordinate context is ignored. In GeoConnect, motion was associated with task duration, attempts, errors, and hesitation under participant-clustered inference; HMD path, for example, was strongly associated with stage duration ($\rho=.748$, 95\% CI [0.674, 0.814]). These findings support coordinate-aware motion telemetry as a complementary source of behavioral evidence for immersive HCI, while emphasizing that inferred motion states must remain grounded in task context rather than treated as universal cognitive labels.Keywords
This publication has 13 references indexed in Scilit:
- Comparing locomotion techniques in virtual reality: Effects on user experience and psychological appraisal of urban environmentComputers in Human Behavior Reports, 2026
- CRAVE: Cross-Reality Analytics for Virtual Environments — deepening insights into human food selection and eating behaviorComputers in Human Behavior Reports, 2026
- Walking experience in real and virtual environments: A comparative studyComputers in Human Behavior Reports, 2026
- Leveraging virtual reality to study risk preference and impulsivity: Insights from a dual scoping reviewComputers in Human Behavior Reports, 2025
- Self-regulated learning in immersive virtual reality: A case study on a pre-service teacher's behavior and perceptionsComputers in Human Behavior Reports, 2025
- Metaverse technologies and human behavior: Insights into engagement, adoption, and ethical challengesComputers in Human Behavior Reports, 2025
- Towards estimating affective states in Virtual Reality based on behavioral dataVirtual Reality, 2021
- The Cognitive Affective Model of Immersive Learning (CAMIL): a Theoretical Research-Based Model of Learning in Immersive Virtual RealityEducational Psychology Review, 2021
- Immersive analysis of user motion in VR applicationsThe Visual Computer, 2020
- Functional performance in a virtual reality task with differential executive functional loadsComputers in Human Behavior Reports, 2020