Distributed anomaly detection using 1‐class SVM for vertically partitioned data
Authors:
- 29 June 2011
- journal article
- research article
- Published by Wiley in Statistical Analysis and Data Mining: An ASA Data Science Journal
- Vol. 4 (4) , 393-406
Abstract
There has been a tremendous increase in the volume of sensor data collected over the last decade for different monitoring tasks. For example, petabytes of earth science data are collected from modern satellites,in situsensors and different climate models. Similarly, huge amount of flight operational data is downloaded for different commercial airlines. These different types of data sets need to be analyzed for finding outliers. Information extraction from such rich data sources using advanced data mining methodologies is a challenging task not only because of the massive volume of data but also because these data sets are physically stored at different geographical locations with only a subset of features available at any location. Moving these petabytes of data to a single location may waste a lot of bandwidth. To solve this problem, in this paper, we present a novel algorithm which can identify outliers in the entire data without moving all the data to a single location. The method we propose only centralizes a very small sample from the different data subsets at different locations. We analytically prove and experimentally verify that the algorithm offers high accuracy compared to complete centralization with only a fraction of the communication cost. We show that our algorithm is highly relevant to both earth sciences and aeronautics by describing applications in these domains. The performance of the algorithm is demonstrated on two large publicly available data sets: (i) the NASA MODIS satellite images and (ii) a simulated aviation data set generated by the ‘Commercial Modular Aero‐Propulsion System Simulation’ (CMAPSS). © 2011 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 4: 393–406, 2011Keywords
This publication has 23 references indexed in Scilit:
- Anomaly detectionACM Computing Surveys, 2009
- A Scalable Local Algorithm for Distributed Multivariate RegressionStatistical Analysis and Data Mining: An ASA Data Science Journal, 2008
- Detecting anomalous records in categorical datasets2007
- Revealing land cover change in California with satellite dataEos, 2007
- A Survey of Outlier Detection MethodologiesArtificial Intelligence Review, 2004
- Mining distance-based outliers in near linear time with randomization and a simple pruning rule2003
- Spatial contextual classification and prediction models for mining geospatial dataIEEE Transactions on Multimedia, 2002
- Estimating the Support of a High-Dimensional DistributionNeural Computation, 2001
- Efficient algorithms for mining outliers from large data setsACM SIGMOD Record, 2000
- Distance-based outliers: algorithms and applicationsThe VLDB Journal, 2000