Advancing Pre-Trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection
- 17 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 4509-4522
Abstract
With the wide application of knowledge distillation between an ImageNet pre-trained teacher model and a learnable student model, unsupervised anomaly detection has witnessed a significant achievement in the past few years. The success of this framework mainly relies on how to keep the feature discrepancy between the teacher and student model, in which it has two underlying sub-assumptions: (1) The teacher model can represent two separable distributions for the normal and abnormal patterns, while (2) the student model can only reconstruct the normal distribution. However, it still remains a challenging issue to maintain these ideal assumptions in practice. In this paper, we propose a simple yet effective two-stage industrial anomaly detection framework, termed AAND, which sequentially performs Anomaly Amplification and Normality Distillation to enhance the two assumptions. In the first anomaly amplification stage, we propose a novel Residual Anomaly Amplification (RAA) module to advance the pre-trained teacher encoder with synthetic anomalies. It generates adaptive residuals to amplify anomalies while maintaining the feature integrity of pre-trained model. It mainly comprises a Matching-guided Residual Gate and an Attribute-scaling Residual Generator, which can determine the residuals’ proportion and characteristic, respectively. In the second normality distillation stage, we further employ a reverse distillation paradigm to train a student decoder, in which a novel Hard Knowledge Distillation (HKD) loss is built to better facilitate the reconstruction of normal patterns. Comprehensive experiments on the MvTecAD, VisA, and MvTec3D-RGB datasets show that our method achieves state-of-the-art performance. Our code is available at https://github.com/Hui-design/AAND.Keywords
All Related Versions
Funding Information
- National Science and Technology Major Project (2023ZD0121300)
- National Natural Science Foundation of China (62572384)
- National Natural Science Foundation of China (U24A20325)
- National Natural Science Foundation of China (62125305)
- National Natural Science Foundation of China (62503384)
- Key Research and Development Plan of Shaanxi Province (2024PT-ZCK-80)
This publication has 45 references indexed in Scilit:
- Neural Metamorphosis2024
- Memory-augmented appearance-motion network for video anomaly detectionPattern Recognition, 2023
- SPot-the-Difference Self-supervised Pre-training for Anomaly Detection and Segmentation2022
- DSR – A Dual Subspace Re-Projection Network for Surface Anomaly Detection2022
- Natural Synthetic Anomalies for Self-supervised Anomaly Detection and Localization2022
- Factorizing Knowledge in Neural Networks2022
- Reconstruction by inpainting for visual anomaly detectionPattern Recognition, 2021
- PaDiM: A Patch Distribution Modeling Framework for Anomaly Detection and Localization2021
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- ImageNet: A large-scale hierarchical image database2009