Guanyi Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Guanyi Li has a total of 36 co-authors. Here are the top co-authors by number of shared publications:
- Dan Zeng
School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China and The Technology Innovation Center for Collaborative Applications of Natural Resources Data in GBA, MNR, Guangzhou, China
8 shared publications - Junjie Zhang
Xian Jiaotong-liverpool University, Suzhou, Jiangsu, China
7 shared publications - Hongwen Yu
Shanghai University
2 shared publications - Rui Gao
School of Communication and Information Engineering, Shanghai University, Shanghai, China
2 shared publications Anqi Ding
Key Laboratory of Specialty Fiber Optics and Optical Access Networks, Joint International Research Laboratory of Specialty Fiber Optics and Advanced Communication, Shanghai Institute of Advanced Communication and Data Science, Shanghai University, Shanghai, China
1 shared publication- Chenyu Dong
Key Laboratory of Specialty Fiber Optics and Optical Access Networks, Joint International Research Laboratory of Specialty Fiber Optics and Advanced Communication, Shanghai Institute of Advanced Communication and Data Science, Shanghai University, Shanghai, China
1 shared publication
Latest Publications
- Open Access
- 9 July 2026
- Published by Elsevier in SSRN Electronic Journal
Infrared UAV Tracking for Low-Altitude Airborne Remote Sensing via Reliability-Aware Local-Global Collaboration
Airborne infrared remote sensing is critical for low-altitude earth observation, yet tracking small, agile targets like unmanned aerial vehicles (UAVs) in remote sensing imagery remains highly challenging. To meet the efficiency constraints of airborne sensing payloads, local tracking paradigms are essential. However, conventional local trackers often unconditionally propagate prior hypotheses, making them vulnerable to search region degradation and cumulative drift under complex sensor-target coupled motion. To address this issue, we propose a reliability-aware local-global collaborative framework for infrared UAV tracking. First, a Reliability-Aware Search Regulation (RASR) mechanism dynamically evaluates hypothesis reliability, adaptively selecting between propagated and motion-compensated search centers to suppress error propagation. Second, a Temporally Conditioned Global Re-initialization (GRI) mechanism selectively triggers full-frame detection only upon sustained degradation, utilizing multi-cue temporal assessment to recover from failures while preserving high throughput. Extensive experiments demonstrate that our method achieves state-of-the-art performance on Anti-UAV410 and Anti-UAV benchmarks, yielding 65.9\% and 69.6\% AUC, respectively. Operating at 32.7 FPS, the proposed framework effectively bridges the accuracy-robustness gap of local trackers in airborne remote sensing applications.
- 9 July 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessInfrared UAV Tracking for Low-Altitude Airborne Remote Sensing via Reliability-Aware Local-Global Collaboration
Airborne infrared remote sensing is critical for low-altitude earth observation, yet tracking small, agile targets like unmanned aerial vehicles (UAVs) in remote sensing imagery remains highly challenging. To meet the efficiency constraints of airborne sensing payloads, local tracking paradigms are essential. However, conventional local trackers often unconditionally propagate prior hypotheses, making them vulnerable to search region degradation and cumulative drift under complex sensor-target coupled motion. To address this issue, we propose a reliability-aware local-global collaborative framework for infrared UAV tracking. First, a Reliability-Aware Search Regulation (RASR) mechanism dynamically evaluates hypothesis reliability, adaptively selecting between propagated and motion-compensated search centers to suppress error propagation. Second, a Temporally Conditioned Global Re-initialization (GRI) mechanism selectively triggers full-frame detection only upon sustained degradation, utilizing multi-cue temporal assessment to recover from failures while preserving high throughput. Extensive experiments demonstrate that our method achieves state-of-the-art performance on Anti-UAV410 and Anti-UAV benchmarks, yielding 65.9\% and 69.6\% AUC, respectively. Operating at 32.7 FPS, the proposed framework effectively bridges the accuracy-robustness gap of local trackers in airborne remote sensing applications.
- 6 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. PP (10577149) , 1
MIST: A Benchmark and Baseline for Multi-frame Infrared Small Target Detection in Complex Motion
Motion cues play a vital role in multi-frame infrared small target detection (MISTD). However, most targets in existing datasets exhibit regular and slow motion, which cannot reflect the complex and diverse motion patterns in real-world scenarios. This biased data distribution makes recent data-driven methods highly rely on simplified motion assumptions that tend to fail in irregular or fast motion, resulting in noisy feature representations cluttered with target-irrelevant factors. Hence, we stress that methods for MISTD should also work when targets are in complex motion. To enable this research, we propose a large-scale dataset called MIST for airborne infrared detection scenarios. The dataset is built on a synthetic data engine that models variations in pose, size, and intensity of moving targets while seamlessly blending them into real backgrounds for physical, geometric, and visual realism. Targets in MIST exhibit low signal-to-clutter ratios and complex motion, making it a promising yet challenging benchmark for developing algorithms focused on motion analysis. To tackle the challenges of MIST, we develop MISTNet, a robust baseline based on the Information Bottleneck theory. To handle irregular and fast motion, we propose a shifted neighborhood compensation block to efficiently model multi-scale correspondences for implicit motion compensation. To distill compact representations free from irrelevant cues, we design a progressive distillation decoder to hierarchically filter out redundancy while preserving target-relevant information. We benchmark 31 state-of-the-art methods and find that their performance on MIST drops significantly compared with that on the widely used NUDT-MIRSDT dataset. Our MISTNet outperforms all other methods by a large margin, with an over 6% gain in the IoU metric, demonstrating its superiority. The dataset, code, and model weights are available at https://github.com/GR-ray/MIST.
- 6 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. PP (10577149) , 1
MIST: A Benchmark and Baseline for Multi-frame Infrared Small Target Detection in Complex Motion
Motion cues play a vital role in multi-frame infrared small target detection (MISTD). However, most targets in existing datasets exhibit regular and slow motion, which cannot reflect the complex and diverse motion patterns in real-world scenarios. This biased data distribution makes recent data-driven methods highly rely on simplified motion assumptions that tend to fail in irregular or fast motion, resulting in noisy feature representations cluttered with target-irrelevant factors. Hence, we stress that methods for MISTD should also work when targets are in complex motion. To enable this research, we propose a large-scale dataset called MIST for airborne infrared detection scenarios. The dataset is built on a synthetic data engine that models variations in pose, size, and intensity of moving targets while seamlessly blending them into real backgrounds for physical, geometric, and visual realism. Targets in MIST exhibit low signal-to-clutter ratios and complex motion, making it a promising yet challenging benchmark for developing algorithms focused on motion analysis. To tackle the challenges of MIST, we develop MISTNet, a robust baseline based on the Information Bottleneck theory. To handle irregular and fast motion, we propose a shifted neighborhood compensation block to efficiently model multi-scale correspondences for implicit motion compensation. To distill compact representations free from irrelevant cues, we design a progressive distillation decoder to hierarchically filter out redundancy while preserving target-relevant information. We benchmark 31 state-of-the-art methods and find that their performance on MIST drops significantly compared with that on the widely used NUDT-MIRSDT dataset. Our MISTNet outperforms all other methods by a large margin, with an over 6% gain in the IoU metric, demonstrating its superiority. The dataset, code, and model weights are available at https://github.com/GR-ray/MIST.
- Open Access
- 6 April 2026
- Published by Elsevier in SSRN Electronic Journal
From Part to Whole: Structural Semantic Knowledge Guided Airplane Detection in Remote Sensing Images
Airplane detection in remote sensing images, which aims to localize and recognize airplanes in aerial or satellite imagery, is of great importance due to its broad applications in both civil and military scenarios. However, most existing methods primarily rely on holistic object-level representations and therefore struggle to capture the underlying structural relationships when airplanes are only partially visible. To tackle this problem, we propose a novel two-stage plug-and-play structural semantic knowledge guided framework that explicitly models part-to-whole structural relationships for airplane detection, consisting of Semantic Knowledge Generation followed by Part Guided Detection. In the first stage, a Structural Semantic Extraction Module (SSEM) is introduced to capture the semantic characteristics and model the relationships between the whole object and its parts, and then forms an airplane structural semantic knowledge pool. In the second stage, the structural semantics from the knowledge pool is leveraged by an attention-based Part Semantic Aggregation Module (PSAM) to guide the detector to perceive part-to-whole structural relationships, thereby improving feature representations. Extensive experiments on DOTA-v1.0-Plane, DOTA-v1.5-Plane, and FAIR1M-v2.0-Plane demonstrate that the proposed framework consistently improves detection performance, particularly for partially visible airplanes. The code will be publicly released upon acceptance.
- 6 April 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessFrom Part to Whole: Structural Semantic Knowledge Guided Airplane Detection in Remote Sensing Images
Airplane detection in remote sensing images, which aims to localize and recognize airplanes in aerial or satellite imagery, is of great importance due to its broad applications in both civil and military scenarios. However, most existing methods primarily rely on holistic object-level representations and therefore struggle to capture the underlying structural relationships when airplanes are only partially visible. To tackle this problem, we propose a novel two-stage plug-and-play structural semantic knowledge guided framework that explicitly models part-to-whole structural relationships for airplane detection, consisting of Semantic Knowledge Generation followed by Part Guided Detection. In the first stage, a Structural Semantic Extraction Module (SSEM) is introduced to capture the semantic characteristics and model the relationships between the whole object and its parts, and then forms an airplane structural semantic knowledge pool. In the second stage, the structural semantics from the knowledge pool is leveraged by an attention-based Part Semantic Aggregation Module (PSAM) to guide the detector to perceive part-to-whole structural relationships, thereby improving feature representations. Extensive experiments on DOTA-v1.0-Plane, DOTA-v1.5-Plane, and FAIR1M-v2.0-Plane demonstrate that the proposed framework consistently improves detection performance, particularly for partially visible airplanes. The code will be publicly released upon acceptance.
- 20 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28 (15209210) , 360-372
Learning Compact Representations With an Information Bottleneck for Camouflaged Object Detection
Frequency domain-based methods have demonstrated promising performance in Camouflaged Object Detection (COD) tasks because of their enhanced power for distinguishing between objects and the background in the frequency domain. However, these methods often overlook the interference caused by task-irrelevant cues such as background textures. These extraneous factors are learned alongside task-relevant features by the employed network, increasing the number of false positives. Therefore, we propose a camouflaged object detection method based on the Information Bottleneck (IB) theory. The aim is to obtain a robust representation that retains the essential features needed for prediction while minimizing the redundant information derived from both the RGB and frequency domains. Specifically, we propose a Feature Selection Information Bottleneck Module (FSIBM). By explicit supervision, this module minimizes the mutual information between the fused feature from two domains and the predictive features, thereby weakening task-irrelated information. Simultaneously, the FSIBM maximizes the mutual information between the predictive features and the ground truth (i.e., emphasizing task-related elements). Additionally, we introduce a Cross-Domain Awareness Interaction Module (CDAIM), which establishes self-reinforcement for the object attributes within each domain and facilitates cross-domain complementarity. This enables the capture of sufficient discriminative features from both domains. To verify the generalization ability of the proposed method, we applied it to three benchmark datasets, on which our method outperformed the corresponding state-of-the-art methods. Our code is released at https://github.com/KwunYat/CODIB.
- 20 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28 (15209210) , 360-372
Learning Compact Representations With an Information Bottleneck for Camouflaged Object Detection
Frequency domain-based methods have demonstrated promising performance in Camouflaged Object Detection (COD) tasks because of their enhanced power for distinguishing between objects and the background in the frequency domain. However, these methods often overlook the interference caused by task-irrelevant cues such as background textures. These extraneous factors are learned alongside task-relevant features by the employed network, increasing the number of false positives. Therefore, we propose a camouflaged object detection method based on the Information Bottleneck (IB) theory. The aim is to obtain a robust representation that retains the essential features needed for prediction while minimizing the redundant information derived from both the RGB and frequency domains. Specifically, we propose a Feature Selection Information Bottleneck Module (FSIBM). By explicit supervision, this module minimizes the mutual information between the fused feature from two domains and the predictive features, thereby weakening task-irrelated information. Simultaneously, the FSIBM maximizes the mutual information between the predictive features and the ground truth (i.e., emphasizing task-related elements). Additionally, we introduce a Cross-Domain Awareness Interaction Module (CDAIM), which establishes self-reinforcement for the object attributes within each domain and facilitates cross-domain complementarity. This enables the capture of sufficient discriminative features from both domains. To verify the generalization ability of the proposed method, we applied it to three benchmark datasets, on which our method outperformed the corresponding state-of-the-art methods. Our code is released at https://github.com/KwunYat/CODIB.
- 17 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Geoscience and Remote Sensing Letters
- Vol. 22 (1545598X) , 1-5
Learning Instructive Frequency Spectral and Curvature Features for Cloud Detection
Current cloud detection methods often treat all spectral bands equally, which limits their ability to capture instructive clues necessary for accurate detection. As a result, distinguishing clouds from snow in coexisting environments remains challenging. Moreover, most approaches struggle to adaptively model the boundaries of clouds, which is crucial for detecting thin clouds with ambiguous edges. To address these challenges, we propose a novel approach for cloud detection called FSCFNet, which captures guiding visual features from frequency and curvature computations. FSCFNet comprises two key modules: the Frequency Spectral Feature Enhancement Module (FSFEM) and the Curvature-based Edge-Awareness Module (CEAM). The FSFEM leverages the distinct characteristics of spectral bands to extract instructive visual cues, enabling the network to learn robust discriminative features for ice, snow, and clouds. In contrast, the CEAM adaptively identifies texture-rich regions using curvature, enhancing the ability to delineate thin cloud boundaries. Comprehensive quantitative and qualitative experiments on the Landsat 8 and MODIS datasets demonstrate that FSCFNet consistently outperforms state-of-the-art methods. Our code is publicly available at https://github.com/wanjuanhu/FSCFNet/tree/main.
- 17 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Geoscience and Remote Sensing Letters
- Vol. 22 (1545598X) , 1-5
Learning Instructive Frequency Spectral and Curvature Features for Cloud Detection
Current cloud detection methods often treat all spectral bands equally, which limits their ability to capture instructive clues necessary for accurate detection. As a result, distinguishing clouds from snow in coexisting environments remains challenging. Moreover, most approaches struggle to adaptively model the boundaries of clouds, which is crucial for detecting thin clouds with ambiguous edges. To address these challenges, we propose a novel approach for cloud detection called FSCFNet, which captures guiding visual features from frequency and curvature computations. FSCFNet comprises two key modules: the Frequency Spectral Feature Enhancement Module (FSFEM) and the Curvature-based Edge-Awareness Module (CEAM). The FSFEM leverages the distinct characteristics of spectral bands to extract instructive visual cues, enabling the network to learn robust discriminative features for ice, snow, and clouds. In contrast, the CEAM adaptively identifies texture-rich regions using curvature, enhancing the ability to delineate thin cloud boundaries. Comprehensive quantitative and qualitative experiments on the Landsat 8 and MODIS datasets demonstrate that FSCFNet consistently outperforms state-of-the-art methods. Our code is publicly available at https://github.com/wanjuanhu/FSCFNet/tree/main.
- Open Access
- 1 May 2024
- Published by MDPI AG in Applied Sciences
- Vol. 14 (9) , 3890
Adaptive Shadow Compensation Method in Hyperspectral Images via Multi-Exposure Fusion and Edge Fusion
Shadows in hyperspectral images lead to reduced spectral intensity and changes in spectral characteristics, significantly hindering analysis and applications. However, current shadow compensation methods face the issue of nonlinear attenuation at different wavelengths and unnatural transitions at the shadow boundary. To address these challenges, we propose a two-stage shadow compensation method based on multi-exposure fusion and edge fusion. Initially, shadow regions are identified through color space conversion and an adaptive threshold. The first stage utilizes multi-exposure, generating a series of exposure images through adaptive exposure coefficients that reflect spatial shadow intensity variations. Fusion weights for exposure images are determined based on exposure, contrast, and spectral variance. Then, the exposure sequence and fusion weights are constructed as Laplacian pyramids and Gaussian pyramids, respectively, to obtain a weighted fused exposure sequence. In the second stage, the previously identified shadow regions are smoothly reintegrated into the original image using edge fusion based on the p-Laplacian operator. To further validate the effectiveness and spectral fidelity of our method, we introduce a new hyperspectral image dataset. Experimental results on the public dataset and proposed dataset demonstrate that our method surpasses other mainstream shadow compensation methods.
- 1 May 2024
- Published by MDPI AG in Applied Sciences
- Vol. 14 (9) , 3890
Open AccessAdaptive Shadow Compensation Method in Hyperspectral Images via Multi-Exposure Fusion and Edge Fusion
Shadows in hyperspectral images lead to reduced spectral intensity and changes in spectral characteristics, significantly hindering analysis and applications. However, current shadow compensation methods face the issue of nonlinear attenuation at different wavelengths and unnatural transitions at the shadow boundary. To address these challenges, we propose a two-stage shadow compensation method based on multi-exposure fusion and edge fusion. Initially, shadow regions are identified through color space conversion and an adaptive threshold. The first stage utilizes multi-exposure, generating a series of exposure images through adaptive exposure coefficients that reflect spatial shadow intensity variations. Fusion weights for exposure images are determined based on exposure, contrast, and spectral variance. Then, the exposure sequence and fusion weights are constructed as Laplacian pyramids and Gaussian pyramids, respectively, to obtain a weighted fused exposure sequence. In the second stage, the previously identified shadow regions are smoothly reintegrated into the original image using edge fusion based on the p-Laplacian operator. To further validate the effectiveness and spectral fidelity of our method, we introduce a new hyperspectral image dataset. Experimental results on the public dataset and proposed dataset demonstrate that our method surpasses other mainstream shadow compensation methods.
- 28 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (9) , 8412-8426
Multi-Level Information Fusion Network With Edge Information Injection for Single-Band Cloud Detection
Current cloud detection methods have demonstrated effectiveness by utilizing the rich spectral features of multi-spectral images. Compared to multispectral images, single-band infrared images offer higher efficiency in terms of sampling and processing speed. However, single-band cloud detection methods have not been fully developed, and existing methods based on multispectral cloud detection have some limitations when applied directly to single-band images: Firstly, they often blend shallow features containing spatial details with deep features providing high-level semantic information, yet struggle to disentangle features with strong discrimination representing cloud edges and bodies from limited information. Additionally, the correlation between features at different aspects is not fully reasoned, resulting in blurred boundary segmentation. To address these issues, we introduce a Multi-level Information Fusion Network (MIFNet) with an integrated edge information injection strategy. Our method effectively decouples clouds into their fundamental components: body and edge (Low-Frequency (LF) and High-Frequency (HF) components), enabling the comprehensive acquisition of strong discriminative features. Specifically, we propose an Edge Feature Extraction Module (EFEM) that isolates the cloud body through low-pass filtering, while the cloud’s edge is extracted by subtracting lower-level features from LF components. Furthermore, we employ a Feature Refinement Module (FRM) to locate the cloud body’s position precisely. Building upon this foundation, we devise a Graph Reasoning Module (GRM) to facilitate the full inference of feature correlations at different levels and to model the global interdependence between edges and semantics. Through comprehensive evaluations on benchmark datasets comprising infrared band images from Landsat 8 and MODIS satellites, we demonstrate that our proposed MIFNet outperforms state-of-the-art methods, yielding promising results in cloud detection accuracy. Our code is publicly available at https://github.com/KwunYat/MIFNet.
- 28 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (9) , 8412-8426
Multi-Level Information Fusion Network With Edge Information Injection for Single-Band Cloud Detection
Current cloud detection methods have demonstrated effectiveness by utilizing the rich spectral features of multi-spectral images. Compared to multispectral images, single-band infrared images offer higher efficiency in terms of sampling and processing speed. However, single-band cloud detection methods have not been fully developed, and existing methods based on multispectral cloud detection have some limitations when applied directly to single-band images: Firstly, they often blend shallow features containing spatial details with deep features providing high-level semantic information, yet struggle to disentangle features with strong discrimination representing cloud edges and bodies from limited information. Additionally, the correlation between features at different aspects is not fully reasoned, resulting in blurred boundary segmentation. To address these issues, we introduce a Multi-level Information Fusion Network (MIFNet) with an integrated edge information injection strategy. Our method effectively decouples clouds into their fundamental components: body and edge (Low-Frequency (LF) and High-Frequency (HF) components), enabling the comprehensive acquisition of strong discriminative features. Specifically, we propose an Edge Feature Extraction Module (EFEM) that isolates the cloud body through low-pass filtering, while the cloud’s edge is extracted by subtracting lower-level features from LF components. Furthermore, we employ a Feature Refinement Module (FRM) to locate the cloud body’s position precisely. Building upon this foundation, we devise a Graph Reasoning Module (GRM) to facilitate the full inference of feature correlations at different levels and to model the global interdependence between edges and semantics. Through comprehensive evaluations on benchmark datasets comprising infrared band images from Landsat 8 and MODIS satellites, we demonstrate that our proposed MIFNet outperforms state-of-the-art methods, yielding promising results in cloud detection accuracy. Our code is publicly available at https://github.com/KwunYat/MIFNet.
- 26 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Geoscience and Remote Sensing Letters
- Vol. 21 (1545598X) , 1-5
Leveraging Frequency-Guided Mixer and Target-Aware Attention for Ground-Based Cloud Detection
Compared to satellite imagery, ground-based cameras capture cloud data (ground-to-sky data) with higher temporal and spatial resolutions, providing more detailed cloud information. However, the spectral information available in ground-to-sky data is limited. Therefore, extracting features with strong discrimination from optical remote sensing images (ORSIs) is challenging. Currently, deep learning-based cloud detection methods face two main challenges. Firstly, although Convolutional Neural Networks (CNNs) effectively extract high-frequency (HF) components from images through convolutions, they struggle to capture low-frequency (LF) components, which are capable of representing global features and target structures. Secondly, in ORSIs, the spectral characteristics of thin clouds and the sky are similar, making it difficult to distinguish cloud regions from the background. To address these challenges, we propose a network consisting of two main modules: the Mixer Module (MM) and the Cloud Aware Attention Module (CAAM). The MM comprises a HF and a LF components extraction branch. The HF branch extracts local textures through max-pooling and parallel convolution operations. The LF branch captures long-range dependency by decomposing a large kernel convolution. It leverages the advantages of both convolution and self-attention to effectively capture global features. In addition, we introduce the CAAM, which quantifies images into histograms to separate clouds from the background and enhances the perception of clouds using attention mechanism. We conducted experiments using both daytime and nighttime cloud image data from the SWINySeg dataset with mIoU reaching 88.93% and OA reaching 93.97%. The results demonstrate that our proposed method achieves promising performance compared to state-of-the-art cloud detection methods.
- 26 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Geoscience and Remote Sensing Letters
- Vol. 21 (1545598X) , 1-5
Leveraging Frequency-Guided Mixer and Target-Aware Attention for Ground-Based Cloud Detection
Compared to satellite imagery, ground-based cameras capture cloud data (ground-to-sky data) with higher temporal and spatial resolutions, providing more detailed cloud information. However, the spectral information available in ground-to-sky data is limited. Therefore, extracting features with strong discrimination from optical remote sensing images (ORSIs) is challenging. Currently, deep learning-based cloud detection methods face two main challenges. Firstly, although Convolutional Neural Networks (CNNs) effectively extract high-frequency (HF) components from images through convolutions, they struggle to capture low-frequency (LF) components, which are capable of representing global features and target structures. Secondly, in ORSIs, the spectral characteristics of thin clouds and the sky are similar, making it difficult to distinguish cloud regions from the background. To address these challenges, we propose a network consisting of two main modules: the Mixer Module (MM) and the Cloud Aware Attention Module (CAAM). The MM comprises a HF and a LF components extraction branch. The HF branch extracts local textures through max-pooling and parallel convolution operations. The LF branch captures long-range dependency by decomposing a large kernel convolution. It leverages the advantages of both convolution and self-attention to effectively capture global features. In addition, we introduce the CAAM, which quantifies images into histograms to separate clouds from the background and enhances the perception of clouds using attention mechanism. We conducted experiments using both daytime and nighttime cloud image data from the SWINySeg dataset with mIoU reaching 88.93% and OA reaching 93.97%. The results demonstrate that our proposed method achieves promising performance compared to state-of-the-art cloud detection methods.
- 6 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 7823-7837
Frequency-Aware Multi-Modal Fine-Tuning for Few-Shot Open-Set Remote Sensing Scene Classification
Few-shot open-set recognition, as a new paradigm, leveraging a limited amount of supervised data to identify specific Remote Sensing (RS) scene categories and generalize to novel ones. However, the data bias induced by the small sample size not only causes severe overfitting within base classes, but also impairs the capacity for inference to identify RS scenes in hitherto unobserved categories. Furthermore, owing to environmental influences, RS images frequently manifest notable intra-class disparities and comparatively low inter-class distinctions, intensifying the challenge in obtaining suitable classifiers. To address above issues, we investigate the utilization of a Multi-modal Foundational Model (MFM) infused with essential domain knowledge to mitigate the generalization limitations encountered in few-shot scenarios. Recognizing that existing MFMs with a visual-text dual-branch structure are primarily tailored for natural scenes, we propose a custom Frequency Distribution-based Multi-modal Fine-Tuning strategy (FreqDiMFT) in a parameter-efficient manner. More specifically, within the vision branch, we address the high inter-class similarity and intra-class diversity in RS images by embedding the local-global frequency distribution information to facilitate the recognition of RS scenes. To further amplify the model's generalization ability post transfer, we introduce an adaptive feature refinement module designed for Transformers, proficient in filtering redundant features resulting from domain disparities. To mitigate the domain drift on the textual branch, we adopt an input format that combines basic templates with domain expertise from RS end to generate more discriminative class prototypes. To fully verify the effectiveness of our FreqDiMFT in a more practical setting, we collect a Large-Scale hybrid dataset (LSRS). Extensive experiments demonstrate that, even with a scant number of training samples, our strategy yields advanced performances compared to state-of-the-art models.
- 6 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 7823-7837
Frequency-Aware Multi-Modal Fine-Tuning for Few-Shot Open-Set Remote Sensing Scene Classification
Few-shot open-set recognition, as a new paradigm, leveraging a limited amount of supervised data to identify specific Remote Sensing (RS) scene categories and generalize to novel ones. However, the data bias induced by the small sample size not only causes severe overfitting within base classes, but also impairs the capacity for inference to identify RS scenes in hitherto unobserved categories. Furthermore, owing to environmental influences, RS images frequently manifest notable intra-class disparities and comparatively low inter-class distinctions, intensifying the challenge in obtaining suitable classifiers. To address above issues, we investigate the utilization of a Multi-modal Foundational Model (MFM) infused with essential domain knowledge to mitigate the generalization limitations encountered in few-shot scenarios. Recognizing that existing MFMs with a visual-text dual-branch structure are primarily tailored for natural scenes, we propose a custom Frequency Distribution-based Multi-modal Fine-Tuning strategy (FreqDiMFT) in a parameter-efficient manner. More specifically, within the vision branch, we address the high inter-class similarity and intra-class diversity in RS images by embedding the local-global frequency distribution information to facilitate the recognition of RS scenes. To further amplify the model's generalization ability post transfer, we introduce an adaptive feature refinement module designed for Transformers, proficient in filtering redundant features resulting from domain disparities. To mitigate the domain drift on the textual branch, we adopt an input format that combines basic templates with domain expertise from RS end to generate more discriminative class prototypes. To fully verify the effectiveness of our FreqDiMFT in a more practical setting, we collect a Large-Scale hybrid dataset (LSRS). Extensive experiments demonstrate that, even with a scant number of training samples, our strategy yields advanced performances compared to state-of-the-art models.