Integrating Cross-Modal Semantic Learning with Generative Models for Gesture Recognition
Open Access
Abstract
Radio frequency (RF)-based human activity sensing is an essential component of ubiquitous computing, with WiFi sensing providing a practical and low-cost solution for gesture and activity recognition. However, challenges such as manual data collection, multipath interference, and poor cross-domain generalization hinder real-world deployment. Existing data augmentation approaches often neglect the biomechanical structure underlying RF signals. To address these limitations, we present CM-GR, a cross-modal gesture recognition framework that integrates semantic learning with generative modeling. CM-GR leverages 3D skeletal points extracted from vision data as semantic priors to guide the synthesis of realistic WiFi signals, thereby incorporating biomechanical constraints without requiring extensive manual labeling. In addition, dynamic conditional vectors are constructed from inter-subject skeletal differences, enabling user-specific WiFi data generation without the need for dedicated data collection and annotation for each new user. Extensive experiments on the public MM-Fi dataset and our SelfSet dataset demonstrate that CM-GR substantially improves the cross-subject gesture recognition accuracy, achieving gains of up to 10.26% and 9.5%, respectively. These results confirm the effectiveness of CM-GR in synthesizing personalized WiFi data and highlight its potential for robust and scalable gesture recognition in practical settings.Keywords
Funding Information
- National Natural Science Foundation of China (62406296, 202203021222049, 202303021212193, 20230017)
- Natural Science Basic Research Program of Shanxi Province (62406296, 202203021222049, 202303021212193, 20230017)
- Program for the Scientific Activities of Selected Returned Overseas Professionals in Shanxi Province (62406296, 202203021222049, 202303021212193, 20230017)
This publication has 48 references indexed in Scilit:
- Navigating the Realm of Generative Models: GANs, Diffusion, Limitations, and Future Prospects—A Review2024
- Challenges and solutions for vision-based hand gesture interpretation: A reviewComputer Vision and Image Understanding, 2024
- Computer vision-based hand gesture recognition for human-robot interaction: a reviewComplex & Intelligent Systems, 2023
- A survey on application in RF signalMultimedia Tools and Applications, 2023
- GAN-based sensor data augmentation: Application for counting moving people and detecting directions using PIR sensorsEngineering Applications of Artificial Intelligence, 2022
- Gesture recognition from RGB images using convolutional neural network‐attention based systemConcurrency and Computation: Practice and Experience, 2022
- U2-Net: Going deeper with nested U-structure for salient object detectionPattern Recognition, 2020
- Dynamic gesture recognition by using CNNs and star RGB: A temporal information condensationNeurocomputing, 2020
- What Makes Training Multi-Modal Classification Networks Hard?2020
- 3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training2019