Multimodal Industrial Anomaly Detection via Uni-Modal and Cross-Modal Fusion
- 3 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 21 (6) , 5000-5010
Abstract
Constructing comprehensive multimodal feature representations from RGB images (RGB) and point clouds (PT) in 2D–3D multimodal anomaly detection (MAD) methods is very important to reveal various types of industrial anomalies. For multimodal representations, most of the existing MAD methods often consider the explicit spatial correspondence between the modality-specific features extracted from RGB and PT through space-aligned fusion, while overlook the implicit interaction relationships between them. In this study, we propose a uni-modal and cross-modal fusion (UCF) method, which comprehensively incorporates the implicit relationships within and between modalities in multimodal representations. Specifically, UCF first establishes uni-modal and cross-modal embeddings to capture intramodal and intermodal relationships through uni-modal reconstruction and cross-modal mapping. Then, an adaptive nonequal fusion method is proposed to develop fusion embeddings, with the aim of preserving the primary features and reducing interference of the uni-modal and cross-modal embeddings. Finally, uni-modal, cross-modal, and fusion embeddings are all collaborated to reveal anomalies existing in different modalities. Experiments conducted on the MVTec 3D-AD benchmark and the real-world surface mount inspection demonstrate that the proposed UCF outperforms existing approaches, particularly in precise anomaly localization.Keywords
Funding Information
- National Natural Science Foundation of China (62271214)
- Natural Science Foundation of Guangdong Province (2024A1515011466)
- Science and Technology Program of Guangzhou (2023B01J0037)
This publication has 27 references indexed in Scilit:
- Complementary pseudo multimodal feature for point cloud anomaly detectionPattern Recognition, 2024
- Masked Autoencoders for Point Cloud Self-supervised Learning2022
- Towards Total Recall in Industrial Anomaly Detection2022
- Anomaly Detection via Reverse Distillation from One-Class Embedding2022
- Reconstruction by inpainting for visual anomaly detectionPattern Recognition, 2021
- PaDiM: A Patch Distribution Modeling Framework for Anomaly Detection and Localization2021
- Unsupervised anomaly segmentation via deep feature reconstructionNeurocomputing, 2021
- Uninformed Students: Student-Teacher Anomaly Detection With Discriminative Latent Embeddings2020
- MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection2019
- Deep Residual Learning for Image Recognition2016