Coordinate-Aware Visual Analytics of Head–Hand Motion Across Immersive Interaction Regimes
Authors:
- Fabiana PeresFPFabiana Peres
- João Marcelo TeixeiraJTJoão Marcelo Teixeira
- Fátima L. S. NunesFNFátima L. S. Nunes
Preprint
- 18 August 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Immersive virtual reality and mixed reality applications continuously produce rich motion telemetry from head-mounted displays, hands, controllers, and interaction devices. However, these data are often reduced to aggregate task outcomes or discarded after the experiment, limiting their value for understanding how users behave across different immersive tasks. This paper proposes a coordinate-aware framework for transforming HMD and hand/controller telemetry into interpretable temporal motion signatures. The framework extracts global, window-based, relative-motion, input, and coordination features, while explicitly distinguishing world-space, head-relative, and task-relative interpretations of movement. A browser-based visual analytics interface links overview, temporal,paired-condition, performance-grounded, and data-quality views, allowing analysts to inspect motion signatures from cross-application summaries down to individual session timelines. We evaluate the framework across four immersive interaction regimes. The first is FAST, a full-scale VR assembly dataset with 216 sessions in which users learn and manipulate tubes, connectors, screws, and tools to assemble two structures. The second is a physically grounded MR/VR driving experiment with 80 pose-telemetry runs collected while participants sat inside a real BYD Dolphin Plus and controlled a virtual vehicle using the real steering wheel and pedals; in the present artifact, BYD is used for coordinate-frame and seated-control motion analysis. The third is GeoConnect, an educational cartography application with 36 HMD session files, yielding 206 performance-linked map-board stages from 35 contributing sessions involving completion time, attempts, errors, precision, and hesitation. The fourth is Logic Gate, an educational virtual-laboratory application with 177 sessions in which users assemble logical expressions by selecting components, placing them on a table, and connecting circuit inputs and outputs. Across the four regimes, the framework exposes distinct motion organizations. FAST exhibited a threshold-robust bimanual-centered temporal pattern, with 80.6\% of analyzed windows classified as bimanual. In BYD, world-space HMD paths exceeded 3~km despite participants remaining physically seated, illustrating the importance of coordinate-aware interpretation. In GeoConnect, motion descriptors were significantly associated with stage-level performance variables under participant-clustered inference; HMD path, for example, was strongly associated with duration ($\rho=.748$, 95\% CI [0.674, 0.814]). Logic Gate exhibited planning-like low-activity intervals, inspection-consistent head motion, and asymmetric and bimanual constructive-interaction patterns.Keywords
This publication has 11 references indexed in Scilit:
- Introducing VAMPIRE – Using Kinaesthetic Feedback in Virtual Reality for Automated Driving Experiments2022
- A study of how immersion and interactivity drive VR learningComputers & Education, 2022
- Towards estimating affective states in Virtual Reality based on behavioral dataVirtual Reality, 2021
- The Cognitive Affective Model of Immersive Learning (CAMIL): a Theoretical Research-Based Model of Learning in Immersive Virtual RealityEducational Psychology Review, 2021
- Immersive analysis of user motion in VR applicationsThe Visual Computer, 2020
- A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agendaComputers & Education, 2020
- Automated Usability Evaluation of Virtual Reality ApplicationsACM Transactions on Computer-Human Interaction, 2019
- The VR Book2015
- How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User PresenceMedia Psychology, 2014
- Place illusion and plausibility can lead to realistic behaviour in immersive virtual environmentsPhilosophical Transactions of the Royal Society B: Biological Sciences, 2009