Toward a Foundation Model for Time Series Data
Open Access
- 21 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4400-4404
Abstract
A foundation model is a machine learning model trained on a large and diverse set of data, typically using self-supervised learning-based pre-training techniques, that can be adapted to various downstream tasks. However, current research on time series pre-training has predominantly focused on models trained exclusively on data from a single domain. As a result, these models possess domain-specific knowledge that may not be easily transferable to time series from other domains. In this paper, we aim to develop an effective time series foundation model by leveraging unlabeled samples from multiple domains. To achieve this, we repurposed the publicly available UCR Archive and evaluated four existing self-supervised learning-based pre-training methods, along with a novel method, on the datasets. We tested these methods using four popular neural network architectures for time series to understand how the pre-training methods interact with different network designs. Our experimental results show that pre-training improves downstream classification tasks by enhancing the convergence of the fine-tuning process. Furthermore, we found that the proposed pre-training method, when combined with the Transformer, outperforms the alternatives. The proposed method outperforms or achieves equal performance compared to the second best method in ~93% of downstream tasks.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- A Survey on Vision TransformerIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Mixing up contrastive learning: Self-supervised representation learning for time seriesPattern Recognition Letters, 2022
- An Empirical Study of Training Self-Supervised Vision Transformers2021
- A Transformer-based Framework for Multivariate Time Series Representation Learning2021
- An empirical survey of data augmentation for time series classification with neural networksPLOS ONE, 2021
- GHashing: Semantic Graph Hashing for Approximate Similarity Search in Graph Databases2020
- Learning to Hash with Graph Neural Networks for Recommender Systems2020
- The UCR time series archiveIEEE/CAA Journal of Automatica Sinica, 2019
- Deep Residual Learning for Image Recognition2016
- Searching and mining trillions of time series subsequences under dynamic time warping2012