Adaptive Bernstein change detector for high-dimensional data streams
Open Access
- 9 January 2024
- journal article
- research article
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 38 (3) , 1334-1363
Abstract
Change detection is of fundamental importance when analyzing data streams. Detecting changes both quickly and accurately enables monitoring and prediction systems to react, e.g., by issuing an alarm or by updating a learning algorithm. However, detecting changes is challenging when observations are high-dimensional. In high-dimensional data, change detectors should not only be able to identify when changes happen, but also in which subspace they occur. Ideally, one should also quantify how severe they are. Our approach, ABCD, has these properties. ABCD learns an encoder-decoder model and monitors its accuracy over a window of adaptive size. ABCD derives a change score based on Bernstein’s inequality to detect deviations in terms of accuracy, which indicate changes. Our experiments demonstrate that ABCD outperforms its best competitor by up to 20% in F1-score on average. It can also accurately estimate changes’ subspace, together with a severity measure that correlates with the ground truth.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (Research Training Group GRK 2153)
- Karlsruher Institut für Technologie (KIT)
This publication has 48 references indexed in Scilit:
- Self-Adaptive Windowing Approach for Handling Complex Concept DriftCognitive Computation, 2015
- Detecting concept change in dynamic data streamsMachine Learning, 2014
- A Nonparametric Approach for Multiple Change Point Analysis of Multivariate DataJournal of the American Statistical Association, 2014
- Optimal Detection of Changepoints With a Linear Computational CostJournal of the American Statistical Association, 2012
- Leveraging Bagging for Evolving Data Streams2010
- Learning from Time-Changing Data with Adaptive Windowing2007
- Critical values for multiple structural change testsThe Econometrics Journal, 2003
- Updating Formulae and a Pairwise Algorithm for Computing Sample Variances1982
- CONTINUOUS INSPECTION SCHEMESBiometrika, 1954
- Economic Quality Control of Manufactured Product1Bell System Technical Journal, 1930