ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions
Top Cited Papers
- 16 March 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 35 (12) , 12181-12193
Abstract
Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical-Cumulative-distribution-based Outlier Detection), which is inspired by the fact that outliers are often the “rare events” that appear in the tails of a distribution. In a nutshell, ECOD first estimates the underlying distribution of the input data in a nonparametric fashion by computing the empirical cumulative distribution per dimension of the data. ECOD then uses these empirical distributions to estimate tail probabilities per dimension for each data point. Finally, ECOD computes an outlier score of each data point by aggregating estimated tail probabilities across dimensions. Our contributions are as follows: (1) we propose a novel outlier detection method called ECOD , which is both parameter-free and easy to interpret; (2) we perform extensive experiments on 30 benchmark datasets, where we find that ECOD outperforms 11 state-of-the-art baselines in terms of accuracy, efficiency, and scalability; and (3) we release an easy-to-use and scalable (with distributed support) Python implementation for accessibility and reproducibility.Keywords
All Related Versions
This publication has 42 references indexed in Scilit:
- Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the medianJournal of Experimental Social Psychology, 2013
- A New Algorithm for Detecting Outliers in Linear RegressionInternational Journal of Statistics and Probability, 2013
- A comparison of outlier detection algorithms for ITS dataExpert Systems with Applications, 2010
- Anomaly detectionACM Computing Surveys, 2009
- Outlier Detection with Globally Optimal Exemplar-Based GMM2009
- Discovering cluster-based local outliersPattern Recognition Letters, 2003
- Estimating the Support of a High-Dimensional DistributionNeural Computation, 2001
- Efficient algorithms for mining outliers from large data setsACM SIGMOD Record, 2000
- Measuring Skewness and KurtosisJournal of the Royal Statistical Society: Series D (The Statistician), 1984
- Optimal Rates of Convergence for Nonparametric EstimatorsThe Annals of Statistics, 1980