Evaluation-free Time-series Forecasting Model Selection via Meta-learning
Open Access
- 11 March 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 19 (3) , 1-41
Abstract
Time-series forecasting models are invariably used in a variety of domains for crucial decision-making. Traditionally these models are constructed by experts with considerable manual effort. Unfortunately, this approach has poor scalability while generating accurate forecasts for new datasets belonging to diverse applications. Without access to skilled domain-knowledge, one approach is to train all the models on the new time-series data and then select the best one. However, this approach is nonviable in practice. In this work, we develop techniques for fast automatic selection of the best forecasting model for a new unseen time-series dataset, without having to first train (or evaluate) all the models on the new time-series data to select the best one. In particular, we develop a forecasting meta-learning approach called AutoForecast that allows for the quick inference of the best time-series forecasting model for an unseen dataset. Our approach learns both forecasting models’ performances over time horizon of the same dataset and task similarity across different datasets. The experiments demonstrate the effectiveness of the approach over state-of-the-art (SOTA) single and ensemble methods and several SOTA meta-learners (adapted to our problem) in terms of selecting better forecasting models (i.e., 2 \(\times\) gain) for unseen tasks for univariate and multivariate testbeds. AutoForecast has also significant reduction in inference time compared to the naïve approach (doing inference using all possible models and then selecting the best one), with median of 42 \(\times\) across the two testbeds. We release our meta-learning database corpus (348 datasets), performances of the 322 forecasting models on the database corpus, meta-features, and source codes for the community to access them for forecasting model selection and to build on them with new datasets and models which can help advance automating time-series forecasting problem. In our released database corpus, we unveil new traces of Adobe computing cluster usage for production workloads.
Keywords
This publication has 53 references indexed in Scilit:
- Meta‐learning how to forecast time seriesJournal of Forecasting, 2023
- Evaluating time series forecasting models: an empirical study on performance estimation methodsMachine Learning, 2020
- catch22: CAnonical Time-series CHaracteristicsData Mining and Knowledge Discovery, 2019
- Transfer learning with seasonal and trend adjustment for cross-building energy forecastingEnergy and Buildings, 2018
- Scalable Gaussian process-based transfer surrogates for hyperparameter optimizationMachine Learning, 2017
- Speeding up algorithm selection using average ranking and active testing by introducing runtimeMachine Learning, 2017
- A Bayesian ridge regression analysis of congestion's impact on urban expressway safetyAccident Analysis & Prevention, 2016
- Simple algorithm portfolio for SATArtificial Intelligence Review, 2011
- Prediction of multivariate time series by autoregressive model fittingJournal of Multivariate Analysis, 1985
- Exponential smoothing: The state of the artJournal of Forecasting, 1985