Junhui Li

In the field of bridging communication gaps between deaf and hearing communities, sign language recognition (SLR) and sign language translation (SLT) systems face persistent challenges in cross-lingual contexts as the understanding ability of unlabeled data is limited, the fusion depth of traditional fusion modes is insufficient, and inadequate temporal modeling, which hinder practical performance. To address these limitations, this study proposes TriFuseNet, an innovative open-vocabulary tri-modal fusion framework grounded in intersemiotic translation theory, dynamic equivalence theory, and multimodal discourse analysis, which integrates RGB images, textual semantics and skeletal keypoints through a hierarchical interaction mechanism to achieve fine-grained cross-modal alignment. The framework introduces a vision-language interaction module to bridge visual features with linguistic structures, while a bilinear state-space model (BSSM) captures long-range temporal dynamics for gesture-linguistic coherence. Additionally, region-specific attention mechanisms prioritize critical body regions to model culturally specific gestures with nuanced spatial-temporal details. Experimental evaluations on the CSL-Daily and PHOENIX-14T datasets demonstrate TriFuseNet’s superiority over state-of-the-art methods, achieving 2.3% and 1.6% reductions in word error rate (WER), respectively.

Dear Editor, This letter proposes an innovative open-vocabulary 3D scene understanding model based on visual-language model. By efficiently integrating 3D point cloud data, image data, and text data, our model effectively overcomes the segmentation problem [1], [2] of traditional models dealing with unknown categories [3]. By deeply learning the deep semantic mapping between vision and language, the network significantly improves its ability to recognize unlabeled categories and exceeds current state-of-the-art methods in the task of scene understanding in open-vocabulary.

Hearing impairments affect approximately 1.5 billion individuals worldwide, highlighting the critical need for effective communication tools between deaf and hearing populations. Traditional sign language translation (SLT) models predominantly rely on gloss-based methods, which convert visual sign language inputs into intermediate gloss sequences before generating textual translations. However, these methods are constrained by their reliance on extensive annotated data, susceptibility to error propagation, and inadequate handling of low-frequency or unseen sign language vocabulary, thus limiting their scalability and practical application. Drawing upon multimodal translation theory, this study proposes the open-vocabulary sign language translation (OVSLT) method, designed to overcome these challenges by integrating open-vocabulary principles. OVSLT introduces two pivotal modules: Enhanced Caption Generation and Description (CGD), and Grid Feature Grouping with Advanced Alignment Techniques. The Enhanced CGD module employs a GPT model enhanced with a Negative Retriever and Semantic Retrieval-Augmented Features (SRAF) to produce semantically rich textual descriptions of sign gestures. In parallel, the Grid Feature Grouping module applies Grid Feature Grouping, contrastive learning, feature-discriminative contrastive loss, and balanced region loss scaling to refine visual feature representations, ensuring robust alignment with textual descriptions. We evaluated OVSLT on the PHOENIX-14T and CSLDaily datasets. The results demonstrated a ROUGE score of 29.6% on the PHOENIX-14T dataset and 30.72% on the CSLDaily dataset, significantly outperforming existing models. These findings underscore the versatility and effectiveness of OVSLT, showcasing the potential of open-vocabulary approaches to surpass the limitations of traditional SLT systems and contribute to the evolving field of multimodal translation.

The open-vocabulary understanding of UAV aerial images plays a crucial role in enhancing the intelligence level of remote sensing applications, such as disaster assessment, precision agriculture, and urban planning. In this paper, we propose an innovative open-vocabulary model for UAV images, which combines vision–language methods to achieve efficient recognition and segmentation of unseen categories by generating multi-view image descriptions and feature extraction. To enhance the generalization ability and robustness of the model, we adopted Mixup technology to blend multiple UAV images, generating more diverse and representative training data. To address the limitations of existing open-vocabulary models in UAV image analysis, we leverage the GPT model to generate accurate and professional text descriptions of aerial images, ensuring contextual relevance and precision. The image encoder utilizes a U-Net with Mamba architecture to extract key point information through edge detection and partition pooling, further improving the effectiveness of feature representation. The text encoder employs a fine-tuned BERT model to convert text descriptions of UAV images into feature vectors. Three key loss functions were designed: Generalization Loss to balance old and new category scores, semantic segmentation loss to evaluate model performance on UAV image segmentation tasks, and Triplet Loss to enhance the model’s ability to distinguish features. The Comprehensive Loss Function integrates these terms to ensure robust performance in complex UAV segmentation tasks. Experimental results demonstrate that the proposed method has significant advantages in handling unseen categories and achieving high accuracy in UAV image segmentation tasks, showcasing its potential for practical applications in diverse aerial imagery scenarios.

Accurate classification of three-dimensional (3D) point clouds in real-world environments is often impeded by sensor noise, occlusions, and incomplete data. To overcome these challenges, we propose SMCNet, a robust multimodal framework for 3D point cloud classification. SMCNet combines multi-view projection and neural radiance fields (NeRFs) to generate high-fidelity 2D representations with enhanced texture realism, addressing occlusions and lighting inconsistencies effectively. The Mamba model is further refined within this framework by integrating a depth perception module to capture long-range point interactions and adopting a dual-channel structure to enhance point-wise feature extraction. Fine-tuning adapters for the CLIP and Mamba models are also introduced, significantly improving cross-domain adaptability. Additionally, an intelligent voting mechanism aggregates predictions from multiple viewpoints, ensuring enhanced classification robustness. Comprehensive experiments demonstrate that SMCNet achieves state-of-the-art performance, outperforming the PointNet++ baseline with a 0.5% improvement in mean overall accuracy (mOA) on ModelNet40 and a 7.9% improvement on ScanObjectNN. In corruption resistance, SMCNet reduces the mean corruption error (mCE) by 0.8% on ModelNet40-C and 3.6% on ScanObjectNN-C. These results highlight the effectiveness of SMCNet in tackling real-world classification scenarios with noisy and corrupted data.

In recent years, deep neural networks have achieved significant success in 3D point cloud recognition tasks. However, these models still demonstrate substantial performance challenges in the presence of data corruption. It is crucial to improve the robustness and generalization ability of the model. In this work, we propose a novel framework that combines GPT and CLIP models to enhance the robustness of existing point cloud classification models, which has two main modules: the Text-Image Fusion Module, which includes a GPT-Driven TextGen Processor and FocalView Projection, and the Dual-Path Intelligent Adapter Module. First, the GPT-Driven TextGen Processor leverages GPT-4’s capabilities to generate detailed textual descriptions tailored to point cloud intricacies. FocalView Projection dynamically selects viewpoints based on attention maps, enhancing two-dimensional representations of three-dimensional point clouds. Secondly, the Dual-Path Intelligent Adapter Module achieves fine-tuning and feature adaptation by combining internal and external adapters. Additionally, during the fine-tuning process, we employ a variant of Projected Gradient Descent (PGD) adversarial training, named VPGD, to increase the model’s resilience to adversarial perturbations. Our approach has achieved state-of-the-art results on robust 3D points cloud recognition datasets such as ModelNet40-C and ScanObjectNN-C.