Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
Open Access
- 24 August 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2560-2571
Abstract
Time Series Representation Learning (TSRL) focuses on generating informative representations for various Time Series (TS) modeling tasks. Traditional Self-Supervised Learning (SSL) methods in TSRL fall into four main categories: reconstructive, adversarial, contrastive, and predictive, each with a common challenge of sensitivity to noise and intricate data nuances. Recently, diffusion-based methods have shown advanced generative capabilities. However, they primarily target specific application scenarios like imputation and forecasting, leaving a gap in leveraging diffusion models for generic TSRL. Our work, Time Series Diffusion Embedding (TSDE), bridges this gap as the first diffusion-based SSL TSRL approach. TSDE segments TS data into observed and masked parts using an Imputation-Interpolation-Forecasting (IIF) mask. It applies a trainable embedding function, featuring dual-orthogonal Transformer encoders with a crossover mechanism, to the observed part. We train a reverse diffusion process conditioned on the embeddings, designed to predict noise added to the masked part. Extensive experiments demonstrate TSDE's superiority in imputation, interpolation, forecasting, anomaly detection, classification, and clustering. We also conduct an ablation study, present embedding visualizations, and compare inference speed, further substantiating TSDE's efficiency and validity in learning representations of TS data.Keywords
All Related Versions
Funding Information
- MOTIE (20023495)
This publication has 33 references indexed in Scilit:
- SAITS: Self-attention-based imputation for time seriesExpert Systems with Applications, 2023
- TimeCLR: A self-supervised contrastive learning framework for univariate time series representationKnowledge-Based Systems, 2022
- A multibranch CNN-BiLSTM model for human activity recognition using wearable sensor dataThe Visual Computer, 2021
- Handling incomplete heterogeneous data using VAEsPattern Recognition, 2020
- Transductive LSTM for time-series prediction: An application to weather forecastingNeural Networks, 2020
- Deep Clustering for Unsupervised Learning of Visual Features2018
- Recurrent Neural Networks for Multivariate Time Series with Missing ValuesScientific Reports, 2018
- Gaussian Processes in Machine Learning2004
- Comparing partitionsJournal of Classification, 1985
- Objective Criteria for the Evaluation of Clustering MethodsJournal of the American Statistical Association, 1971