Lei Chen

Transformer-based models have significantly advanced image captioning through self-attention mechanisms and parallel computation. However, existing methods typically adopt teacher-forcing strategies during training by conditioning the decoder exclusively on ground-truth tokens, whereas at inference, captions are generated autoregressively based solely on previously predicted tokens. Such discrepancy between training and inference conditions leads to a progressive accumulation of prediction errors, resulting in captions that deviate significantly from the visual content. While scheduled sampling strategies mitigate this issue, directly integrating them disrupts Transformer parallelism and overlooks visual saliency differences. To tackle these limitations, we present VMFormer, a multi-stage decoding Transformer featuring a Visual-Aware Scheduled Sampling (VASS) module that bridges this gap through two key innovations: 1) A two-stage decoding scheme where an initial self-study stage generates candidate tokens, followed by a hybrid stage dynamically blending ground-truth references with predictions via a visual clues controlled gate. 2) A cognitive-inspired prioritization mechanism that retains visual keywords (nouns/verbs/attributes) in early training phases before transitioning to linguistic refinements, mirroring human captioning patterns. Crucially, the VASS module preserves the parallel computational strengths of Transformer architectures and is designed as a plug-and-play component, readily adaptable to Transformer-based captioning models. Experiments on challenging MSCOCO achieve 142.2 CIDEr. We extend VMFormer to video captioning and demonstrate consistent improvements on MSRVTT and MSVD datasets.

Image recognition for crop diseases using a novel multi-attention module

Deep learning has shown great potential in smart agriculture, especially in the field of pest recognition. However, existing methods require large datasets and do not exploit the semantic associations between multimodal data. To address these problems, this paper proposes a multimodal fine-grained transformer (MMFGT) model, a novel pest recognition method that improves three aspects of transformer architecture to meet the needs of few-shot pest recognition. On the one hand, the MMFGT uses self-supervised learning to extend the transformer structure to extract target features using contrastive learning to reduce the reliance on data volume. On the other hand, fine-grained recognition is integrated into the MMFGT to focus attention on finely differentiated areas of pest images to improve recognition accuracy. In addition, the MMFGT further improves the performance in pest recognition by using the joint multimodal information from the pest’s image and natural language description. Extensive experimental results demonstrate that the MMFGT obtains more competitive results compared to other excellent models, such as ResNet, ViT, SwinT, DINO, and EsViT, in pest recognition tasks, with recognition accuracy up to 98.12% and achieving 5.92% higher accuracy compared to the state-of-the-art DINO method for the baseline.

Impact of dataset on the study of crop disease image recognition

Answer selection is the most important module of question answering system. Mining the semantic relevance between questions and answers is the key point of answer selection. However, the answer texts often contain some colloquial expressions and redundant information in some fields. In these cases, some traditional methods of mining text features are complex and the performance improvement is limited. In this paper we investigate using deep learning to enhance the computability of semantics and eliminate the semantic gap for the answer selection module of non-factual question answering system. By integrating different networks to mine the association semantic between questions and answers, a novel multi-layer answer selection model is proposed. The shallow semantic relevance between questions and answers is mined by adding the attention mechanism to CNN, and then the features are input into Bi-LSTM model for further mining to obtain more efficient semantic representation. The experimental results show that compared with other models, our method can achieve the better accuracy in the task of answer selection.