Mamba‐Enhanced Graph Convolutional Network for Skeleton‐Based Stereoscopic Hand Gesture Recognition in Augmented Reality

Authors:
Publisher website
Google Scholar
Abstract
Skeleton‐based hand gesture recognition in augmented reality (AR) environments demands models that are both accurate and computationally efficient to support real‐time stereoscopic interaction. Existing graph convolutional network (GCN) approaches construct dynamic adjacency matrices primarily through simple pairwise subtraction between joints, which constrains their capacity to capture the nuanced spatiotemporal dependencies inherent in stereoscopic AR gesture data. Furthermore, many current architectures impose substantial parameter overhead, thereby limiting deployment on resource‐constrained AR devices. To address these challenges, this paper proposes a Mamba‐Enhanced Graph Convolutional Network (ME‐GCN), which integrates the selective state space model (SSM) Mamba into a lightweight graph convolutional framework for stereoscopic hand gesture recognition. Specifically, we design a Spatial Graph Mamba Module (SGMM) that leverages the linear‐complexity temporal scanning capability of Mamba to model long‐range inter‐frame dependencies, and a Stereoscopic Depth‐Aware Topology Refinement (SDTR) mechanism that dynamically adjusts the graph topology using binocular depth cues. A Channel‐Temporal Dual‐Stream Fusion (CTDF) strategy further integrates spatial, temporal, and depth features into a unified representation. Extensive experiments conducted on the SHREC'17 Track, DHG‐14/28, and a custom AR‐Stereo‐Gesture dataset demonstrate that ME‐GCN achieves competitive recognition accuracy while maintaining significantly fewer parameters and lower inference latency compared to state‐of‐the‐art methods. Specifically, ME‐GCN attains recognition accuracies of 97.38% and 94.16% on the SHREC'17 Track 14‐class and 28‐class tasks, and 94.28% and 92.14% on DHG‐14/28, while requiring only 0.68 M parameters and 1.08 GFLOPs—a 48.5% parameter reduction relative to TD‐GCN—at an average inference latency of 11.52 ms, the lowest among all compared methods. Future research will extend ME‐GCN to continuous gesture recognition in streaming AR scenarios, multi‐modal RGB‐depth fusion, and lightweight model‐compression techniques for real‐time on‐device AR deployment.

This publication has 24 references indexed in Scilit: