Qi Wang

Accurate and high-throughput variety identification of sorghum kernels is essential for intelligent agriculture and industrial quality control. However, unimodal methods based on either RGB imaging or near-infrared spectroscopy (NIRS) often reach a performance ceiling under subtle inter-variety appearance differences, similar physicochemical compositions, and varying acquisition conditions. In this study, we synchronously collected NIRS spectra (901.467–1706.76 nm, 256 variables) and high-resolution RGB images for six representative sorghum varieties, establishing a paired multimodal dataset with 2,400 spectrum–image samples per variety (14,400 kernels in total) and a unified evaluation protocol (8:2 split with 5-fold cross-validation on training data). We propose an end-to-end Multi-modal Attention Network (MAN) that performs feature-level deep fusion via a bi-directional cross-attention mechanism, enabling adaptive cross-modal interaction (”spectrum↔image”) rather than static concatenation or decision-level aggregation. The spectral branch adopts a 1D-CNN with squeeze-and-excitation (SE) attention, while the image branch leverages transfer learning backbones (ResNet-18/50 and Swin Transformer). Experiments show that MAN consistently outperforms the best unimodal baselines: with a Swin Transformer visual branch, MAN achieves an accuracy of 0.9888 (F1 = 0.9889), outperforming the best NIRS-only baseline (1D-CNN: accuracy 0.9048, F1 0.9102) by 8.40 percentage points in accuracy and 7.72 percentage points in F1, and improving upon the best RGB-only baseline (Swin-T: accuracy 0.9279, F1 0.9143) by 6.09 and 7.31 percentage points, respectively. Ablation studies further suggest that cross-attention accounts for the majority of the gain (early fusion: 0.9493 vs. one-way cross-attention: 0.9768), while bi-directional cross-attention combined with SE yields the best overall performance. Overall, MAN effectively leverages the complementarity between RGB appearance cues and NIRS-derived physicochemical signatures, providing a robust and nondestructive solution for fine-grained sorghum variety identification.

Compared to standard convolution, Winograd algorithm has lower time complexity and can accelerate the execution of convolutional neural networks. Previous studies have utilized Winograd to implement 2D convolution on GPUs, mainly using 2D Winograd, and arranging tensors in NCHW or CHWN format instead of NHWC to make data access coalesced. Due to the higher space complexity of Winograd and limited hardware resources, these implementations are usually confined to small filters. To provide an efficient and flexible fused-Winograd convolution for NHWC format on GPUs, we propose Im2col-Winograd. This algorithm decomposes an ND convolution into a series of 1D convolutions to utilize 1D Winograd, thereby reducing space complexity and data-access discontinuity. The reduced space complexity makes Im2col-Winograd less restricted by hardware capability, enabling it to accommodate a wider range of filter shapes. Our implementations support 2-9 filter widths and do not use any workspace to store intermediate variables. According to the experiments, Im2col-Winograd achieves a speedup of 0.788 × to 2.05 × over the fastest benchmark algorithm in cuDNN; and shows similar convergence to PyTorch on Cifar10 and ILSVRC2012 datasets. Along with memory efficiency, the more generalized acceleration offered by Im2col-Winograd can be beneficial for extracting features at different convolution scales.