Action recognition based on multimode fusion for VR online platform
- 24 February 2023
- journal article
- research article
- Published by Springer Nature in Virtual Reality
- Vol. 27 (3) , 1797-1812
Abstract
The current popular online communication platforms can convey information only in the form of text, voice, pictures, and other electronic means. The richness and reliability of information is not comparable to traditional face-to-face communication. The use of virtual reality (VR) technology for online communication is a viable alternative to face-to-face communication. In the current VR online communication platform, users are in a virtual world in the form of avatars, which can achieve “face-to-face” communication to a certain extent. However, the actions of the avatar do not follow the user, which makes the communication process less realistic. Decision-makers need to make decisions based on the behavior of VR users, but there are no effective methods for action data collection in VR environments. In our work, three modalities of nine actions from VR users are collected using a virtual reality head-mounted display (VR HMD) built-in sensors, RGB cameras and human pose estimation. Using these data and advanced multimodal fusion action recognition networks, we obtained a high accuracy action recognition model. In addition, we take advantage of the VR HMD to collect 3D position data and design a 2D key point augmentation scheme for VR users. Using the augmented 2D key point data and VR HMD sensor data, we can train action recognition models with high accuracy and strong stability. In data collection and experimental work, we focus our research on classroom scenes, and the results can be extended to other scenes.Keywords
Funding Information
- Fundamental Research Funds for the Central Universities (2019CDYGZD004)
- National Natural Science Foundation of China (61876026)
This publication has 45 references indexed in Scilit:
- Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition2016
- School discipline, school uniforms and academic performanceInternational Journal of Educational Management, 2016
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor2015
- Berkeley MHAD: A comprehensive Multimodal Human Action Database2013
- Teachers’ classroom‐based action researchInternational Journal of Research & Method in Education, 2011
- A flexible new technique for camera calibrationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Virtual meeting in cyberstage1998