Peipei Zhu

Video anomaly detection (VAD) aims to identify rare and unpredictable abnormal events in long video streams under open world conditions. While existing surveys predominantly organize content by model architectures or supervision settings, they often obscure the cross-modal alignment challenges, capability boundaries, and practical limitations of diverse sensory cues. To address this gap, this paper presents a unified review from the perspective of multi-modal perception and modeling. We group the existing literature along the dimensions of information sources and semantic granularity into three main categories. These categories are the video modality, model inferred semantic modalities such as skeletons objects and text, and other physical sensing modalities including audio, event streams, and thermal imaging. We systematically analyze the common modeling approaches perceptual information capabilities and environmental limitations of each modality. Furthermore, we summarize the common paradigms of cross-modal fusion such as early, mid, and late fusion, and compare their interaction mechanisms and dependencies. Crucially, we discuss how practical challenges like alignment errors modality degradation and missingness affect system reliability across different fusion stages and outline potential future directions to address these issues. Overall this review provides a structured reference for building robust multi-modal VAD systems and advancing reliable anomaly understanding in real world scenarios.

Blur detection is a novel method for evaluating image quality in the context of Unmanned Aerial Vehicle (UAV) surveillance and monitoring activities. However, existing methods lack multi-scenario adaptability due to the overreliance of deep learning models on learned priors from limited datasets, reducing their adaptability to unfamiliar conditions. To address this issue, the Denoising Diffusion Implicit Model (DDIM) is integrated with a paradigm named Rule-based Semantic Calibration (RSC) to create Rule-Semantic Generative Calibration Blur Detection (RSGC-BD). This approach generates robust blur detection maps through an iterative calibration process that enhances generalization capabilities. Unlike current Blur Detection (BD) methods, which categorize pixels as blurred or unblurred with a single forward propagation, the suggested approach employs the DDIM-based generative model to create and refine a BD map iteratively. By utilizing the iterative calibration process through RSC to integrate rule-based blur masks into generative semantic results at each step, this model ensures high-precision blur prediction, enhanced multi-scenario adaptability, and significantly improved inference speed. Furthermore, we propose a conversion module, namely the Adaptive RGB-to-Grayscale Conversion Cascade (ARGC-Cascade), to convert RGB images to grayscale through adaptive integration, highlighting blurred regions and improving detection accuracy. This enhancement of blur features is achieved by balancing the spectral channel weights during image conversion. The superior performance of the proposed RSGC-BD approach is validated by extensive tests on four high-resolution BD datasets, including the newly introduced UAV-BD. Source codes are available at: https://github.com/udrs/RSGCBD.

This chapter investigates advanced localization techniques for unmanned aerial vehicles (UAVs), focusing on the challenges and solutions associated with both indoor and outdoor environments. By examining a range of positioning strategies, including the global positioning system (GPS), visible light communication (VLC), and vision-based methods, this chapter presents a comprehensive overview of the current state and future potential of UAV localization. The integration of various sensors and the strategic use of landmarks as reference points are explored to enhance localization accuracy. Moreover, two pioneering prototypes that are designed to mitigate the limitations identified in current technologies are introduced for case study. A modularization approach for VLC systems and an empirical method for vision-based distance estimation employing multi-scale QR codes. This chapter also highlights the role of sensor fusion and artificial intelligence in improving the precision and efficiency of UAV navigation. Through proof-of-concept experiments utilizing photodiodes (PDs), inertial measurement units (IMUs), and cameras, we demonstrate the practical applicability and effectiveness of the discussed methodologies. Finally, we study the practical applicability and effectiveness of the discussed systems and their experimental results. Our investigations illustrate significant superiority in UAV localization technologies, paving the way for expanded applications across diverse industries and complex operational scenarios.

In the field of remote sensing (RS), change detection (CD) methods are critical for analyzing the quality of images shot over various geographical areas, particularly for high-resolution images. However, there are some shortcomings of the widely used Convolutional Neural Networks (CNNs) and Transformers-based CD methods. The former is limited by its insufficient long-range modeling capabilities, while the latter is hampered by its computational complexity. Additionally, the commonly used information-fusion methods for pre- and post-change images often lead to information loss or redundancy, resulting in inaccurate edge detection. To address these issues, we propose an Iterative Mamba Diffusion Change Detection (IMDCD) approach to iteratively integrate various pieces of information and efficiently produce fine-grained CD maps. Specifically, the Swin-Mamba-Encoder (SME) within Mamba-CD (MCD) is employed as a semantic feature extractor, capable of modeling long-range relationships with linear computability. Moreover, we introduce the Variable State Space CD (VSS-CD) module, which extracts abundant CD features by training the matrix parameters within the designed State Space Change Detection (SS-CD). The computed high-dimensional CD feature is integrated into the noise predictor using a novel Global Hybrid Attention Transformer (GHAT) while low-dimensional CD features are utilized to calibrate prior CD results at each iterative step, progressively refining the generated outcomes. IMDCD exhibits a high performance across multiple datasets such as the CDD, WHU, LEVIR, and OSCD, marking a significant advancement in the methodologies within the CD field of RS. The code for this work is available on GitHub.

Existing Super-Resolution (SR) methods are typically trained using bicubic degradation simulations, resulting in unsatisfactory results when applied to remote sensing images that contain a wide variety of object shapes and sizes. The insufficient learning approach reduces the focus of models on critical object regions within the images. As a result, their practical performance is significantly hindered, especially in real-world applications where accuracy in object reconstruction is crucial. In this work, we propose a general learning strategy for SR models based on expert knowledge supervision, named EKS-SR, which can incorporate a few coarse-grained semantic information derived from high-level visual tasks into the SR reconstruction process. It utilizes prior information from three perspectives: regional constraints, feature constraints, and attributive constraints, to guide the model to focus more on the object regions within the images. By integrating these expert knowledge-driven constraints, EKS-SR can enhance the model’s ability to accurately reconstruct object regions and capture the key information needed for practical applications. Importantly, this improvement does not increase the inference time and does not require full annotation of the large-scale datasets, but only a few labels, making EKS-SR both efficient and effective. Experimental results demonstrate that the proposed method can achieve improvements in both reconstruction quality and machine vision analysis performance.

Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Existing works usually tackle this task using adversarial learning and visual concept reward based on reinforcement learning. However, these existing works were only able to learn limited cross-domain information in vision and language domains, which restrains the captioning performance of UIC. Inspired by the success of Vision-Language Pre-Trained Models (VL-PTMs) in this research, we attempt to infer the cross-domain cue information about a given image from the large VL-PTMs for the UIC task. This research is also motivated by recent successes of prompt learning in many downstream multi-modal tasks, including image-text retrieval and vision question answering. In this work, a semantic prompt is introduced and aggregated with visual features for more accurate caption prediction under the adversarial learning framework. In addition, a metric prompt is designed to select high-quality pseudo image-caption samples obtained from the basic captioning model and refine the model in an iterative manner. Extensive experiments on the COCO and Flickr30 K datasets validate the promising captioning ability of the proposed model. We expect that the proposed prompt-based UIC model will stimulate a new line of research for the VL-PTMs based captioning.

Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Existing works usually tackle this task using adversarial learning and visual concept reward based on reinforcement learning. However, these existing works were only able to learn limited cross-domain information in vision and language domains, which restrains the captioning performance of UIC. Inspired by the success of Vision-Language Pre-Trained Models (VL-PTMs) in this research, we attempt to infer the cross-domain cue information about a given image from the large VL-PTMs for the UIC task. This research is also motivated by recent successes of prompt learning in many downstream multi-modal tasks, including image-text retrieval and vision question answering. In this work, a semantic prompt is introduced and aggregated with visual features for more accurate caption prediction under the adversarial learning framework. In addition, a metric prompt is designed to select high-quality pseudo image-caption samples obtained from the basic captioning model and refine the model in an iterative manner. Extensive experiments on the COCO and Flickr30K datasets validate the promising captioning ability of the proposed model. We expect that the proposed prompt-based UIC model will stimulate a new line of research for the VL-PTMs based captioning.

The goal of unpaired image captioning (UIC) is to describe images without using image-caption pairs in the training phase. Although challenging, we except the task can be accomplished by leveraging a training set of images aligned with visual concepts. Most existing studies use off-the-shelf algorithms to obtain the visual concepts because the Bounding Box (BBox) labels or relationship-triplet labels used for the training are expensive to acquire. In order to resolve the problem in expensive annotations, we propose a novel approach to achieve cost-effective UIC. Specifically, we adopt image-level labels for the optimization of the UIC model in a weakly-supervised manner. For each image, we assume that only the image-level labels (such as object categories and the relationships) are available without specific locations and numbers. The image-level labels are utilized to train a weakly-supervised object recognition model to extract object information (e.g., instance) in an image, and the extracted instances are adopted to infer the relationships among different objects based on an enhanced graph neural network (GNN). The proposed approach achieves comparable or even better performance compared with previous methods without the expensive cost of annotations. Furthermore, we design an unrecognized object (UnO) loss combined with a visual concept reward to improve the alignment of the inferred object and relationship information with the images. It can effectively alleviate the issue encountered by existing UIC models about generating sentences with nonexistent objects. To the best of our knowledge, this is the first attempt to solve the problem of Weakly-Supervised visual concept recognition for UIC (WS-UIC) based only on image-level labels. Extensive experiments have been carried out to demonstrate that the proposed WS-UIC model achieves inspiring results on the COCO dataset while significantly reducing the cost of labeling.