Yaoru Sun

In multimodal semantic segmentation tasks of urban street scenes, existing methods lack modeling of intermodal structural alignment and semantic cooperation between architectures, leading to insufficient fusion feature representations. To address this issue, this article proposes a novel structural optimization network: a hybrid embedding segment anything model (SAM) guided feedback network (GFNet). This network is based on the SAM framework and achieves multimodal structural alignment by transforming the semantic prior (SP) extractor through module-level fine-tuning of the image encoder. Furthermore, this article proposes a cross-architecture knowledge transfer (CAKT) mechanism, injecting the structural awareness capability of SAM into the backbone features of each layer, achieving dual optimization of alignment and enhancement. To address the issues of intermodal heterogeneity and semantic conflict, this article combines complementary fusion at different frequencies and cross-modal similarity enhancement strategies to achieve fine-grained semantic fusion and consistency modeling, supplemented by a dual-supervised constraint mechanism to improve modal independence and robustness. On several challenging datasets, mAcc is improved by about 5%, and GFNet demonstrates the superior segmentation performance and robustness compared to existing methods. Our code will be released to the public at https://github.com/WBangG/GFNet

The crowd density estimation has recently attracted attention from various industries. For efficient computing in the real-time applications, this paper proposes a lightweight relation-aware network based on self-contrastive distillation for red‒green‒blue, thermal (RGB-T) crowd counting. In order to better deal with irregular area images taken by drones, a novel multi-relationship graph reasoning module is designed in the network. By modeling and reasoning the graph node relationship on multi-layer, deep interaction in resolution is achieved from the pixel level, and rich multimodal information is obtained. Furthermore, to make full use of the information on self-attention, the gaussian selection module is introduced, in which gaussian filtering is used to obtain valuable information and convert it into a probabilistic form for further optimized modal communication. This paper also proposes a new cyclic contrast distillation training method. By dividing positive samples and negative samples for comparative learning and training, the intermediate features are optimized to influence the final output, and the final output is then transferred to the underlying features for self-distillation. In this way, the intermediate information is strengthened and a new training cycle is formed. The feature expression of each layer is greatly optimized without increasing parameters. Finally, a large number of experiments have shown that our model performs well on the RGB-T crowd dataset, and the relevant code will be made public on https://github.com/WBangG/LRANet.

With the growth of web technology, the semantic web offers a promising framework for online knowledge collaboration. However, trust issues can undermine users' willingness to collaborate, reduce the frequency of interaction and collaboration efficiency. This paper introduces a super node-based trust management model designed to enhance semantic networks by linking nodes through trust relationships. The model exploits the synergistic incentives of similar interest behaviours to achieve a steady construction of trust relationships. We propose a similarity filtering algorithm that calculates the similarity to filter out false, misleading, or unfair information effectively. Through simulations, we compare our model with RRGRET, Surework, and community-based approaches, and the results show that our model has good network properties, while also resisting multiple malicious attacks and guaranteeing collaboration success. This research contributes to optimizing node relationships within semantic networks and strengthening network robustness against interference.

Accurate detection of surface defects on magnetic chips is a necessary and difficult task, especially for small defects, which lack discriminative and robust features due to their small size and weak characteristics. At present, the low accuracy of small defect detection seriously restricts the development of automated visual inspection and needs to be solved urgently. Thus, we propose a novel small defects detection network based on context-weighted aggregation and feature multiscale loop fusion. First, a context-weighted aggregation module (CAM) that enriches feature representations by combining context and attention mechanisms is proposed. We believe that any further improvements will be futile if robust feature representations cannot be obtained. Second, inspired by the mechanism of looking and thinking twice , we aggregate left-right feedback connections into feature pyramids and creatively propose a loop-shaped feature pyramid network (Loop-FPN), enabling multiscale features to be fused up and down and connected left and right. This loop-shaped structure makes the connection of each layer more direct and allows the features at each scale to be fully integrated, which improves the utilization of multiscale features and facilitates the detection of small defects. Finally, we apply the proposed network to practical detection and the results show that our network achieves 97.57% precision, 91.91% recall, and 98.39% AP, which are 0.13%, 1.43%, and 0.99% higher than the current best-performing comparative methods, respectively. Note to Practitioners —Current vision inspection technology has low accuracy and poor stability in small defect detection, which cannot meet the industrial inspection requirements. Small defect detection has become the biggest challenge in the field of visual inspection and has seriously restricted its development. Our proposed network can solve this problem well by mining small defect context, fusing multiscale features and utilizing attention mechanisms, and has been successfully applied in magnetic chip production line. Furthermore, we developed an image acquisition system that can capture defects on all surfaces with high accuracy and without dead space, allowing our network to not only detect small defects on magnetic chips of varying sizes and irregular shapes, but also to adapt to changes in lighting conditions, backgrounds, and viewpoints. Our work provides an effective, reliable, and convenient quality control solution for magnetic chip production.

With the success of self-supervised learning, multimodal foundation models have rapidly adapted a wide range of downstream tasks driven by vision and language (VL) pre-training. State-of-the-art methods achieve impressive performance by pre-training on large-scale datasets. However, bridging the semantic gap between the two modalities remains a non-negligible challenge for VL tasks. In this work, we propose an efficient computation framework for multimodal alignment by introducing a novel visual semantic module to further improve the performance of the VL tasks. Specifically, we propose a flexible model, namely Artificial-Spiking Hierarchical Networks (ASH-Nets), which combines the complementary advantages of Artificial Neural Network (ANN) and Spiking Neural Network (SNN) to enrich visual semantic representations. In particular, a visual concrete encoder and a semantic abstract encoder are constructed to learn continuous and discrete latent variables to enhance the flexibility of semantic encoding. Considering the spatiotemporal properties of SNN modeling, we introduce a contrastive learning method to optimize the inputs of similar samples. This can improve the computational efficiency of the hierarchical network, while the augmentation of hard samples is beneficial to the learning of visual representations. Furthermore, the Spiking to Text Uni-Alignment Learning (STUA) pre-training method is proposed, which only relies on text features to enhance the encoding ability of abstract semantics. We validate the performance on multiple well-established downstream VL tasks. Experiments show that the proposed ASH-Nets achieve competitive results. Our code is available on GitHub (https://github.com/ZSYTJ/ASH-Nets).