Ziquan Wang

Fog and smoke severely degrade visible-light (RGB) images, obscuring scene structures and impairing the safety and accuracy of visual tasks. Thermal infrared (IR) images are less affected by fog and can supply reliable structural information. Intuitively, RGB-IR fusion should help restore haze-induced structural losses in RGB images. However, our study identifies a critical yet overlooked problem: infrared feature marginalization. During training, models tend to rely excessively on RGB features, failing to exploit IR structural information for dense fog removal. Moreover, existing fusion methods use fog-contaminated RGB features to query corresponding infrared structures, but such degraded RGB signals are nearly noise and cannot retrieve valid IR information effectively. To resolve these issues, we propose MIDF-Net, a dual-modal dehazing framework combining mutual information enhancement, differential fog density estimation, and efficient RGB-IR fusion. First, we design a Fog Density-Aware Mutual Information (\textbf{FDA-MI}) loss along with a lightweight Differential Fog Density Estimator (DFDE) to make IR feature appropriate and important. FDA-MI loss adaptively weights the vanilla InfoNCE loss based on spatially varying fog density, which eliminates blind mutual information optimization. With the reliable fog density predictions from DFDE, we further develop a Fog Density-Gated Infrared Structure Query (\textbf{FDG-ISQ}) module for multi-scale dual-modal feature fusion. This module dynamically calibrates the feature query balance between RGB and infrared branches, allowing the network to build robust query representations under dense fog conditions and effectively recover fog-occluded structural details from infrared modalities, thus achieving high-quality dehazing results. Extensive experiments on three synthetic foggy datasets (DroneVehicle, CARTD, VTUAV) and two real-world datasets (M3FD, ESCVehicle) demonstrate that MIDF-Net surpasses state-of-the-art approaches, particularly in dense fog structural recovery, offering a practical solution for foggy visual perception systems. The code is avaliable at \hyperlink{blue}{https://github.com/aresdrw/MIDF-Net.

Aerial image semantic segmentation is a cornerstone technology for low-altitude economy applications, but lightweight models-indispensable for real-time inference on resource-constrained platforms like drones-suffer from limited feature representation capacity. Visual Foundation Models (VFMs), with robust and generalizable visual semantics learned from massive data, offer great potential for enhancing lightweight models via knowledge distillation (KD). However, direct application of VFM-based KD faces three unaddressed challenges: task-irrelevant knowledge interference in VFMs, inherent intra-class variations in aerial imagery, and inadequate long-range dependency modeling of lightweight networks. To tackle these issues, we propose VFM-CAKD, a category-aware knowledge distillation method tailored for lightweight aerial image semantic segmentation. First, we introduce learnable category-aware vectors to filter task-irrelevant features from VFMs, generating Category-Aware Guidance (CAG) to prioritize task-critical semantics. Second, a Category-Aware-Guided Dual-Scale Feature Fusion (CAG-DSFusion) module is designed to activate category-aware representations, extracting stable intra-class features and robust prototypes to mitigate intra-class variations. Third, a Category-Aware-Guided Global Attention (CAG-Global) module aligns teacher-student global attention maps, transferring VFMs' long-range dependency modeling capability. Experiments on drone-view datasets (Aeroscapes, UAVid), aerial remote sensing datasets (Potsdam, Vaihingen), and a disaster-scenario dataset (RescueNet) demonstrate that VFM-CAKD effectively transfers task-relevant VFM knowledge, significantly improving the performance of lightweight segmentation models. The code is avaliable at https://github.com/aresdrw/VFM-CAKD.

Many Unmanned Aerial Vehicles (UAVs) utilize the visible-light images for environmental perception, making them particularly vulnerable to the degradation of image quality caused by haze or fog. This paper presents a comprehensive solution to enhance the UAV vision specifically in collaborative dehazing and semantic segmentation through three contributions: (1) We propose a multi-task learning framework named Dehaze-Seg integrating U-shaped restoration backbone with frequencyspatial feature fusion to jointly optimize dehazing and segmentation tasks,which results into synchronous output of dehazed images and semantic labels; (2) A multi-density foggy image synthesizing method is proposed by employing metric-scale depth estimation and truncated normal distributions for atmospheric parameter sampling. This results into two proposed multi-density benchmark datasets with synthetic fog, namely HazyUAVid and HazyUDD, respectively; (3) A cyclical network training strategy specifically for robust adaptation to varying fog distribution is proposed. Experimental results demonstrate significant improvements over state-of-the-art methods in both dehazing and semantic segmentation tasks, with powerful generalization among UAV images. We also find that our dehazing module still works well in the ground perspective such as road scene. Codes and datasets are available at https://github.com/aresdrw/dehaze-seg.

Aerial image segmentation typically focuses on broad categories rather than fine details. Besides, objects of the same class often exhibit large variations in texture and scale, while long-range pixel dependencies further complicate segmentation. To address these challenges, we propose VFM-CAKD, a category-aware knowledge distillation method that leverages strong Vision Foundation Models (VFMs) to enhance lightweight segmentation models. First, we introduce learnable categoryaware vectors to filter redundant interference in VFM features, generating a Category-Aware Guidance (CAG) token. Next, to handle intra-class variations, we design a Category-Aware-Guided Dual-Scale Feature Fusion (CAG-DSFusion) strategy, which enhances feature fusion by activating category-aware components, producing more robust category prototypes for effective teacher-student knowledge transferring. Additionally, we propose a Category-Aware-Guided Global Attention module (CAG-Global) to align global feature representations between teacher and student networks, facilitating the transferring of longrange dependency modeling. Experiments on drone-view datasets (Aeroscapes and UAVid) and aerial remote sensing datasets (Potsdam and Vaihingen) demonstrate that VFM-CAKD effectively transfers VFM knowledge to improve student model performance. The code is avaliable at https://github.com/aresdrw/VFM-CAKD.

In traditional 3D reconstruction using UAV images, only radiance information, which is treated as a geometric constraint, is used in feature matching, allowing for the restoration of the scene’s structure. After introducing radiance supervision, NeRF can adjust the geometry in the fixed-ray direction, resulting in a smaller search space and higher robustness. Considering the lack of NeRF construction methods for aerial scenarios, we propose a new NeRF point sampling method, which is generated using a UAV imaging model, compatible with a global geographic coordinate system, and suitable for a UAV view. We found that NeRF is optimized entirely based on the radiance while ignoring the direct geometry constraint. Therefore, we designed a radiance correction strategy that considers the incidence angle. Our method can complete point sampling in a UAV imaging scene, as well as simultaneously perform digital surface model construction and ground radiance information recovery. When tested on self-acquired datasets, the NeRF variant proposed in this paper achieved better reconstruction accuracy than the original NeRF-based methods. It also reached a level of precision comparable to that of traditional photogrammetry methods, and it is capable of outputting a surface albedo that includes shadow information.

Novel view synthesis using neural radiance fields (NeRFs) for remote sensing images is important for various applications. Traditional methods often use implicit representations for modeling, which have slow rendering speeds and cannot directly obtain the structure of the 3D scene. Some studies have introduced explicit representations, such as point clouds and voxels, but this kind of method often produces holes when processing large-scale scenes from remote sensing images. In addition, NeRFs with explicit 3D expression are more susceptible to transient phenomena (shadows and dynamic objects) and even plane holes. In order to address these issues, we propose an improved method for synthesizing new views of remote sensing images based on Point-NeRF. Our main idea focuses on two aspects: filling in the spatial structure and reconstructing ray-marching rendering using shadow information. First, we introduce hole detection, conducting inverse projection to acquire candidate points that are adjusted during training to fill the holes. We also design incremental weights to reduce the probability of pruning the plane points. We introduce a geometrically consistent shadow model based on a point cloud to divide the radiance into albedo and irradiance, allowing the model to predict the albedo of each point, rather than directly predicting the radiance. Intuitively, our proposed method uses a sparse point cloud generated with traditional methods for initialization and then builds the dense radiance field. We evaluate our method on the LEVIR_NVS data set, demonstrating its superior performance compared to state-of-the-art methods. Overall, our work provides a promising approach for synthesizing new viewpoints of remote sensing images.

Test-time adaptation (TTA) has been proven to effectively improve the adaptability of deep learning semantic segmentation models facing continuous changeable scenes. However, most of the existing TTA algorithms lack an explicit exploration of domain gaps, especially those based on visual domain prompts. To address these issues, this paper proposes a self-prompt strategy based on uncertainty, guiding the model to continuously focus on regions with high uncertainty (i.e., regions with a larger domain gap). Specifically, we still use the Mean-Teacher architecture with the predicted entropy from the teacher network serving as the input to the prompt module. The prompt module processes uncertain maps and guides the student network to focus on regions with higher entropy, enabling continuous adaptation to new scenes. This is a self-prompting strategy that requires no prior knowledge and is tested on widely used benchmarks. In terms of the average performance, our method outperformed the baseline algorithm in TTA and continual TTA settings of Cityscapes-to-ACDC by 3.3% and 3.9%, respectively. Our method also outperformed the baseline algorithm by 4.1% and 3.1% on the more difficult Cityscapes-to-(Foggy and Rainy) Cityscapes setting, which also surpasses six other current TTA methods.

Robust segmentation in adverse weather conditions is crucial for autonomous driving. However, these scenes struggle with recognition and make annotations expensive, resulting in poor performance. As a result, the Segment Anything Model (SAM) was recently proposed to finely segment the spatial structure of scenes and to provide powerful prior spatial information, thus showing great promise in resolving these problems. However, SAM cannot be applied directly for different geographic scales and non-semantic outputs. To address these issues, we propose SAM-EDA, which integrates SAM into an unsupervised domain adaptation mean-teacher segmentation framework. In this method, we use a “teacher-assistant” model to provide semantic pseudo-labels, which will fill in the holes in the fine spatial structure given by SAM and generate pseudo-labels close to the ground truth, which then guide the student model for learning. Here, the “teacher-assistant” model helps to distill knowledge. During testing, only the student model is used, thus greatly improving efficiency. We tested SAM-EDA on mainstream segmentation benchmarks in adverse weather conditions and obtained a more-robust segmentation model.

The accurate detection of relevant vehicles, pedestrians, and other targets on the road plays a crucial role in ensuring the safety of autonomous driving. In recent years, object detectors based on Transformers or CNNs have achieved excellent performance in the fully supervised paradigm. However, when the trained model is directly applied to unfamiliar scenes where the training data and testing data have different distributions statistically, the model’s performance may decrease dramatically. To address this issue, unsupervised domain adaptive object detection methods have been proposed. However, these methods often exhibit decreasing performance when the gap between the source and target domains increases. Previous works mainly focused on utilizing the style gap to reduce the domain gap while ignoring the content gap. To tackle this challenge, we introduce a novel method called IDI-SCD that effectively addresses both the style and content gaps simultaneously. Firstly, the domain gap is reduced by disentangling it into the style gap and content gap, generating corresponding intermediate domains in the meanwhile. Secondly, during training, we focus on one single domain gap at a time to achieve inter-domain invariance. That is, the content gap is tackled while maintaining the style gap, and vice versa. In addition, the style-invariant loss is used to narrow down the style gap, and the mean teacher self-training framework is used to narrow down the content gap. Finally, we introduce a multiscale fusion strategy to enhance the quality of pseudo-labels, which mainly focus on enhancing the detection performance for extreme-scale objects (very large or very small objects). We conduct extensive experiments on four mainstream datasets of in-vehicle images. The experimental results demonstrate the effectiveness of our method and its superiority over most of the existing methods.

Semantic segmentation based on optical images can provide comprehensive scene information for intelligent vehicle systems, thus aiding in scene perception and decision making. However, under adverse weather conditions (such as fog), the performance of methods can be compromised due to incomplete observations. Considering the success of domain adaptation in recent years, we believe it is reasonable to transfer knowledge from clear and existing annotated datasets to images with fog. Technically, we follow the main workflow of the previous SDAT-Former method, which incorporates fog and style-factor knowledge into the teacher segmentor to generate better pseudo-labels for guiding the student segmentor, but we identify and address some issues, achieving significant improvements. Firstly, we introduce a consistency loss for learning from multiple source data to better converge the performance of each component. Secondly, we apply positional encoding to the features of fog-invariant adversarial learning, strengthening the model’s ability to handle the details of foggy entities. Furthermore, to address the complexity and noise in the original version, we integrate a simple but effective masked learning technique into a unified, end-to-end training process. Finally, we regularize the knowledge transfer in the original method through re-weighting. We tested our SDAT-Former++ on mainstream benchmarks for semantic segmentation in foggy scenes, demonstrating improvements of 3.3%, 4.8%, and 1.1% (as measured by the mIoU) on the ACDC, Foggy Zurich, and Foggy Driving datasets, respectively, compared to the original version.