Siyu Wang

Multimodal object detection is currently a research hotspot in computer vision. However, the fusion of visible and infrared modalities inevitably increases computational complexity, making most high-performance detection models difficult to deploy on resource-constrained UAV edge devices. Although pruning and knowledge distillation are widely used for model compression, applying them independently often leads to an unstable accuracy–efficiency trade-off. Therefore, this paper proposes a hybrid lightweight algorithm named SAMKD, which combines selective activation pruning with masked knowledge distillation in a staged manner to improve efficiency while maintaining detection performance. Specifically, the selective activation network pruning model (SAPM) first reduces redundant computation by dynamically adjusting network weights and the activation state of input data to generate a lightweight student network. Then, the mask binary classification knowledge distillation (MBKD) strategy is introduced to compensate for this degradation by guiding the student network to recover missing representation patterns under masked feature learning. Moreover, MBKD reformulates classification logits into multiple foreground–background binary mappings, effectively alleviating the severe foreground–background imbalance commonly observed in UAV aerial imagery. This paper constructs a multimodal UAV aerial imagery object detection dataset, M2UD-18K, which includes 9 types of targets and over 18,000 pairs. Extensive experiments show that SAMKD performs well on the self-constructed M2UD-18K dataset, as well as the public DroneVehicle dataset, achieving a favorable trade-off between detection accuracy and detection speed.

Detecting small objects from UAV-captured aerial imagery is a critical yet challenging task, hindered by factors such as small object size, complex backgrounds, and subtle inter-class differences. Single-modal methods lack the robustness for all-weather operation, while existing multimodal solutions are often too computationally expensive for deployment on resource-constrained UAVs. To this end, we propose TAF-YOLO, a lightweight and efficient multimodal detection framework designed to balance accuracy and efficiency. First, we propose an early fusion module, the Two-branch Adaptive Fusion Network (TAFNet), which adaptively integrates visible and infrared information at both pixel and channel levels before the feature extractor, maximizing complementary data while minimizing redundancy. Second, we propose a Large Adaptive Selective Kernel (LASK) module that dynamically expands the receptive field using multi-scale convolutions and spatial attention, preserving crucial details of small objects during downsampling. Finally, we present an optimized feature neck architecture that replaces PANet’s bidirectional path with a more efficient top-down pathway. This is enhanced by a Dual-Stream Attention Bridge (DSAB) that injects high-level semantics into low-level features, improving localization without significant computational overhead. On the VEDAI benchmark, TAF-YOLO achieves 67.2% mAP₅₀, outperforming the CFT model by 2.7% and demonstrating superior performance against seven other YOLO variants. Our work presents a practical and powerful solution that enables real-time, all-weather object detection on resource-constrained UAVs.

Detecting small objects from UAV-captured aerial imagery is a critical yet challenging task, hindered by factors such as small object size, complex backgrounds, and subtle inter-class differences. Single-modal methods lack the robustness for all-weather operation, while existing multimodal solutions are often too computationally expensive for deployment on resource-constrained UAVs. To this end, we propose TAF-YOLO, a lightweight and efficient multimodal detection framework designed to balance accuracy and efficiency. First, an early fusion module, the Two-branch Adaptive Fusion Network (TAFNet), which adaptively integrates visible and infrared information at both pixel and channel levels before the feature extractor, maximizing complementary data while minimizing redundancy. Second, a Large Adaptive Selective Kernel (LASK) module that dynamically expands the receptive field using multi-scale convolutions and spatial attention, preserving crucial details of small objects during downsampling. Finally, an optimized feature neck architecture that replaces PANet's bidirectional path with a more efficient top-down pathway. This is enhanced by a Dual-Stream Attention Bridge (DSAB) that injects high-level semantics into low-level features, improving localization without significant computational overhead. On the VEDAI benchmark, TAF-YOLO achieves 67.2% mAP50, outperforming the CFT model by 2.7% and demonstrating superior performance against seven other YOLO variants. Our work presents a practical and powerful solution that enables real-time, all-weather object detection on resource-constrained UAVs.

In remote sensing target detection cases, great challenges are faced when migrating detection models from the visible domain to the infrared domain. Cross-domain migration suffers from problems such as a lack of data annotations in the infrared domain and interdomain feature differences. To improve the detection accuracy attained for infrared images, we propose a novel two-phase distillation training network, (TPDTNet). Specifically, in the first phase, we incorporate a contrastive learning framework to maximize the mutual information between the source and target domains. In addition, we construct a generative model that learns only a unidirectional modality conversion mapping, thereby capturing the associations between their visual contents. The source-domain image is converted to an image with the style of the target domain, thereby achieving image-level domain alignment. The generated image is combined with the source-domain image to form an enhanced domain for cross-modal training. Enhanced domain data is fed into the teacher network to initialize the weights and produce pseudo-labels. Next, to address small remote sensing target detection tasks, we construct a Multi-dimensional Progressive Feature Fusion Detection (MPFD) framework, which initially fuses two adjacent low-level feature maps and then progressively incorporates high-level features to enhance the quality of fusing non-adjacent layer features. Subsequently, a spatial-dimension convolution is integrated into the backbone network. This convolutional operation is embedded following standard convolution to mitigate the loss of detailed features. Finally, a distillation training strategy that utilizes pseudo detection labels to calculate target information. By minimizing the Kullback-Leibler (KL) divergence between the probability maps of the teacher and student networks, the channel activations are transformed into probability distributions, thereby achieving knowledge distillation. The training weights are transferred from the teacher network to the student network to maximize the detection accuracy. Extensive experiments are conducted on three optical-to-infrared datasets, and the experimental results show that our TPDTNet method achieves state-of-the-art results relative to those of the baseline model.

It is well known that the diverse causes of low-light images challenge the adaptability of enhancement algorithms in uncertain environments. Most deep learning-based algorithms only learn single illuminance estimation or mapping relationship, which inhibit the generalization ability of the model. To address this, we propose a novel multi-illumination estimation framework based on ghost imaging theory, dubbed Ghillie. Specifically, we consider low-light enhancement as a re-imaging process for objects in dark scenes. First, the light modulation network (LMN) is designed to modulate a series of estimated lights following a normal light distribution. These lights "illuminate" the low-light image and the enhanced illuminance image can be reconstructed by a differential ghost imaging algorithm. Then, a gradient-guided denoising network (GDN) is constructed to eliminate noise and enhance details. Finally, we employ the color adaption network (CAN) to restore the color degradation. Additionally, a novel mean structural similarity loss (AM-SSIM) is proposed to guide the model to address the uneven image illumination. The qualitative and quantitative experimental results show that our enhanced methods outperform state-of-the-art methods on the vast majority of publicly available datasets.

Ship detection and formation recognition in remote sensing have increasingly garnered attention. However, research remains challenging due to arbitrary orientation, dense arrangement, and the complex background of ships. To enhance the analysis of ship situations in channels, we model the ships as the key points and propose a context-aware DGCN-based ship formation recognition method. First, we develop a center point-based ship detection subnetwork, which employs depth-separable convolution to reduce parameter redundancy and combines coordinate attention with an oriented response network to generate direction-invariant feature maps. The center point of each ship is predicted by regression of the offset, target scale, and angle to realize the ship detection. Then, we adopt the spatial similarity of the ship center points to cluster the ship group, utilizing the Delaunay triangulation method to establish the topological graph structure of the ship group. Finally, we design a context-aware Dense Graph Convolutional Network (DGCN) with graph structure to achieve formation recognition. Experimental results on HRSD2016 and SGF datasets demonstrate that the proposed method can detect arbitrarily oriented ships and identify formations, attaining state-of-the-art performance.

Cross-view geolocalization of unmanned aerial vehicles (UAVs) is a challenging task due to the positional discrepancies and uncertainties in scale and distance between UAVs and satellite views. Existing transformer-based geolocalization methods mainly use encoders to mine image contextual information. However, these methods have some limitations when dealing with scale changes in cross-view images. Therefore, we present an effective transformer-based Siamese network tailored for UAV geolocalization, called GeoFormer. First, an efficient transformer feature extraction network was designed, which utilizes linear attention to reduce the computational complexity and improve the computational efficiency of the network. Among them, we designed an efficient separable perceptron module based on depthwise separable convolution, which can effectively reduce the computational cost while improving the feature representation of the network. Second, we proposed a multiscale feature aggregation module, which deeply fuses salient features at different scales through a feedforward neural network to generate global feature representations with rich semantics, which improves the model's ability to capture image details and represent robust features. Additionally, we designed a semantic-guided region segmentation module, which utilizes a k -modes clustering algorithm to divide the feature map into multiple regions with semantic consistency and performs feature recognition within each semantic region to improve the accuracy of image matching. Finally, we designed a hierarchical reinforcement rotation matching strategy to achieve accurate UAV geolocalization based on the retrieval results of UAV view query satellite images using SuperPoint keypoints extraction and LightGlue rotation matching. According to the experimental results, our method effectively achieves UAV geolocalization.