Huaiwen Zhang

In the rapidly evolving landscape of digital communication, researchers are increasingly focusing on sign language generation as a means to bridge the communication divide for the Deaf and Hard-of-Hearing communities worldwide. Modeling an efficient sign language video generation framework presents two main challenges: ensuring temporal coherence and handling diverse, complex data distributions, especially given the limited availability of annotated sign video data. To address these challenges, we propose SignMoDP, a sign language video generation model that incorporates a mixture of diffusions for universal sign language generation. We improve temporal coherence by integrating the State Space Model (SSM) into the video diffusion framework, ensuring smooth frame transitions and a natural flow of sign language. By utilizing video-to-patch conversion and a multilingual text encoder, the Sign State Space Model efficiently handles long sequences and complex tasks. Additionally, we introduce the Mixture of Diffusion Paths module to manage diverse data distributions, utilizing expert routing strategies for distinct data regions. We propose a Patch Mixture of Experts (Patch MoE), optimized with low-rank strategies, to further ensure robustness and adaptability in sign language video generation. Evaluations on RWTH-2014 , AUTSL, CSL-Daily, WLASL and LSFB benchmarks demonstrate state-of-the-art performance, positioning SignMoDP as a universal solution for scalable, inclusive sign language generation. More details can be found on our project page: https://github.com/mingtiannihao/SignMoDP.

Directly recognizing action based on compressed video shows significant advantages, such as low storage demands, efficient decoding, and fast inference speeds. Existing methods on compressed video achieve promising performance by separately modeling spatial and motion cues and directly fusing recognition results of I-frames and P-frames. However, these approaches overlook the following inherent attributes of compressed videos: 1) Temporal misalignment between I-frames and P-frames impairs the accuracy of action recognition. 2) Spatiotemporal sparsity of compressed video frames severely hinders the semantic modeling of complex actions. 3) Semantic discrepancy between I-frames and P-frames, which capture appearance and motion information respectively, leads to suboptimal performance when they are fused directly. To address these challenges, we propose a Hierarchical Semantics Interaction Network (HSINet) that ensures refined semantic modeling of compressed video through alignment, interaction, and calibration within a hierarchical fusion framework. Specifically, to resolve temporal misalignment, we propose an efficient cross-modal temporal alignment module that fully combines the spatiotemporal information of P-frames and the spatial information of I-frames, and includes both pre-alignment and fine alignment stages. To mitigate semantic degradation caused by sparsity, we propose a cross-modal semantics interaction module to provide the multi-scale semantics interaction between spatial and temporal representation learning and enhance representations’ spatiotemporal awareness. To calibrate the semantic imbalance between I-frames and P-frames, we propose a modal imbalance calibration module that optimizing directional differences via cosine similarity in hyperspherical space. Experiments on HMDB-51, UCF-101, and Kinetics-400 benchmarks, demonstrate the effectiveness of hierarchical semantic interaction for compressed video action recognition.