Jiafei Wu

In existing restoration-oriented Video Frame Interpolation (VFI) approaches, the motion estimation between neighboring frames plays a crucial role. However, the estimation accuracy in existing methods remains a challenge, primarily due to the inherent ambiguity in identifying corresponding areas in adjacent frames for interpolation. Therefore, enhancing accuracy by distinguishing different regions before motion estimation is of utmost importance. In this paper, we introduce a novel solution involving the utilization of open-world segmentation models, e.g., SAM2 (Segment Anything Model2) for frames, to derive Region-Distinguishable Priors (RDPs) in different frames. These RDPs are represented as spatial-varying Gaussian mixtures, distinguishing an arbitrary number of areas with a unified modality. RDPs can be integrated into existing motion-based VFI methods to enhance features for motion estimation, facilitated by our designed play-and-plug Hierarchical Region-aware Feature Fusion Module (HRFFM). HRFFM incorporates RDP into various hierarchical stages of VFI’s encoder, using RDP-guided Feature Normalization (RDPFN) in a residual learning manner. With HRFFM and RDP, the features within VFI’s encoder exhibit similar representations for matched regions in neighboring frames, thus improving the synthesis of intermediate frames. Extensive experiments demonstrate that HRFFM consistently enhances VFI performance across various scenes.

Most models for Weakly Supervised Video Anomaly Detection (WS-VAD) rely on multiple instance learning, aiming to distinguish normal and abnormal snippets without specifying the type of anomaly. However, the ambiguous nature of anomaly definitions across contexts may introduce inaccuracy in discriminating abnormal and normal events. To show the model what is anomalous, a novel framework is proposed to guide the learning of suspected anomalies from event prompts. Given a textual prompt dictionary of potential anomaly events and the captions generated from anomaly videos, the semantic anomaly similarity between them could be calculated to identify the suspected events for each video snippet. It enables a new multi-prompt learning process to constrain the visual-semantic features across all videos, as well as provides a new way to label pseudo anomalies for self-training. To demonstrate its effectiveness, comprehensive experiments and detailed ablation studies are conducted on four datasets, namely XD-Violence, UCF-Crime, TAD, and ShanghaiTech. Our proposed model outperforms most state-of-the-art methods in terms of AP or AUC (86.5%, 90.4%, 94.4%, and 97.4%). Furthermore, it shows promising performance in open-set and cross-dataset cases. The data, code, and models can be found at: https://github.com/shiwoaz/lap .

Vision-language models have the potential to enrich purely visual tasks by utilizing the combined representation of images/videos and corresponding textual descriptions. Recent advances in video anomaly detection have also integrated textual information to enhance the understanding of abnormal events. However, existing approaches often merge visual and textual modalities in a straightforward, bottom-up manner, failing to fully explore their interconnections. Moreover, textual captions themselves do not inherently convey “abnormal” attributes. Consequently, these joint representations tend to highlight all salient input features without adequately focusing on high-level tasks such as video anomaly detection. To direct the model’s attention towards anomalies more effectively, we propose incorporating a top-down mechanism into weakly supervised video anomaly detection tasks. A new Knowledge Sharing and Feedback (KSF) framework is designed to unify the representation of anomalies across both video and text. Specifically, we develop a category pattern sharing module that performs knowledge matching, acting as an alignment bridge between abnormal events and their corresponding descriptions. This ensures consistent representations for identical anomalies while maintaining distinct representations for different ones. Following this alignment process, matched high-level semantic priors are fed back into the forward path to enhance differentiation between abnormal and normal patterns. Comprehensive experiments on three benchmark datasets demonstrate the superiority of our proposed method in learning the implicit definition of anomaly patterns. The code is available at https://github.com/XJ-Cai/KSF.

Knowledge Distillation (KD) is a widely used model compression technique that primarily transfers knowledge by aligning the predictions of a student model with those of a teacher model. Besides the traditional logit-based KD, combining data augmentation techniques, like MixUp, is another effective way to improve the distillation efficiency. However, as a powerful data augmentation method, PatchMix has shown limited effectiveness in CNN-based knowledge distillation. It is likely due to constraints in the CNN teacher’s receptive field and the absence of PatchMix-retrained teacher models. In this paper, we explore why PatchMix tends to be less effective than MixUp, and further introduce a novel framework called PatchMix Simulation Knowledge Distillation (PS-KD). The proposed new framework simulates a PatchMix-retrained teacher using an vanilla one to guide the student’s training, ensuring the high-fidelity information distillation in feature space. By revisiting the use of PatchMix in CNNs and reducing information distortion, our model is capable to enhance CNN’s spatial invariance and increase the fidelity of network representations. Extensive experiments demonstrate the superiority of our approach, enabling the network to identify discriminative regions in images with greater accuracy. The Code will be released soon.

Pruning neural networks presents a promising avenue for compressing and accelerating contemporary deep convolutional networks. Recently, researchers have increasingly focused on a new kind of stripe-wise pruning (SWP) methods, which offers finer granularity compared to traditional techniques. However, most prior network pruning methods, including SWP, suffer from the issue of “wrong pruning.” During model training, particularly in its early stages, pruning decisions are often sub-optimal, leading to the premature elimination of crucial weights. To mitigate this problem, we propose a novel framework that re-utilizes knowledge from pruned stripe-wise weights for SWP. The unique stripe repair phases are inserted into the standard training process to identify and repair those mis-pruned stripes. Additionally, halfway-pruned models with varying structures are also selectively reused a self-knowledge distillation manner. Comprehensive experiments conducted on three public datasets demonstrate the effectiveness of our proposed model. The code will be released subsequently.