On the need for time series data mining benchmarks
Authors:
Open Access
- 23 July 2002
- conference paper
- Published by Association for Computing Machinery (ACM)
- Vol. 7 (4) , 102-111
Abstract
In the last decade there has been an explosion of interest in mining time series data. Literally hundreds of papers have introduced new algorithms to index, classify, cluster and segment time series. In this work we make the following claim. Much of this work has very little utility because the contribution made (speed in the case of indexing, accuracy in the case of classification and clustering, model accuracy in the case of segmentation) offer an amount of "improvement" that would have been completely dwarfed by the variance that would have been observed by testing on many real world datasets, or the variance that would have been observed by changing minor (unstated) implementation details.To illustrate our point, we have undertaken the most exhaustive set of time series experiments ever attempted, re-implementing the contribution of more than two dozen papers, and testing them on 50 real world, highly diverse datasets. Our empirical results strongly support our assertion, and suggest the need for a set of time series benchmarks and more careful empirical evaluation in the data mining community.Keywords
This publication has 23 references indexed in Scilit:
- Efficient and robust feature extraction and pattern matching of time series by a lattice structure2001
- Segment-based approach for subsequence searches in sequence databases2001
- Discovering similar patterns in time series2000
- Mining the stock market (extended abstract): which measure is best?2000
- Deformable Markov model templates for time-series pattern matching2000
- A fast projection algorithm for sequence data searchingData & Knowledge Engineering, 1998
- MALM1998
- Supporting fast search in time series for movement patterns in multiple scales1998
- Matching and indexing sequences of different lengths1997
- A quantitative study of experimental neural network learning algorithm evaluation practices1995