Interpretable Anomaly Detection with DIFFI: Depth-based Isolation Forest Feature Importance
Preprint
Abstract
Anomaly Detection is an unsupervised learning task aimed at detecting anomalous behaviours with respect to historical data. In particular, multivariate Anomaly Detection has an important role in many applications thanks to the capability of summarizing the status of a complex system or observed phenomenon with a single indicator (typically called `Anomaly Score') and thanks to the unsupervised nature of the task that does not require human tagging. The Isolation Forest is one of the most commonly adopted algorithms in the field of Anomaly Detection, due to its proven effectiveness and low computational complexity. A major problem affecting Isolation Forest is represented by the lack of interpretability, an effect of the inherent randomness governing the splits performed by the Isolation Trees, the building blocks of the Isolation Forest. In this paper we propose effective, yet computationally inexpensive, methods to define feature importance scores at both global and local level for the Isolation Forest. Moreover, we define a procedure to perform unsupervised feature selection for Anomaly Detection problems based on our interpretability method; such procedure also serves the purpose of tackling the challenging task of feature importance evaluation in unsupervised anomaly detection. We assess the performance on several synthetic and real-world datasets, including comparisons against state-of-the-art interpretability techniques, and make the code publicly available to enhance reproducibility and foster research in the field.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- A scale-adaptive positive selection algorithm based on B-cell immune mechanisms for anomaly detectionEngineering Applications of Artificial Intelligence, 2020
- Online and Unsupervised Anomaly Detection for Streaming Data Using an Array of Sliding Windows and PDDsIEEE Transactions on Cybernetics, 2019
- A semisupervised autoencoder-based approach for anomaly detection in high performance computing systemsEngineering Applications of Artificial Intelligence, 2019
- Data-Driven Anomaly Recognition for Unsupervised Model-Free Fault Detection in Artificial PancreasIEEE Transactions on Control Systems Technology, 2018
- Distributed Online One-Class Support Vector Machine for Anomaly Detection Over NetworksIEEE Transactions on Cybernetics, 2018
- Iterative random forests to discover predictive and stable high-order interactionsProceedings of the National Academy of Sciences, 2018
- Anomaly Detection Based on Zone Partition for Security Protection of Industrial Cyber-Physical SystemsIEEE Transactions on Industrial Electronics, 2017
- SafeDrive: Online Driving Anomaly Detection From Large-Scale Vehicle DataIEEE Transactions on Industrial Informatics, 2017
- MediBoost: a Patient Stratification Tool for Interpretable Decision Making in the Era of Precision MedicineScientific Reports, 2016
- Hyperspectral Anomaly Detection by Graph Pixel SelectionIEEE Transactions on Cybernetics, 2015