Probabilistic demand forecasting at scale
Authors:
Open Access
- 1 August 2017
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 10 (12) , 1694-1705
Abstract
We present a platform built on large-scale, data-centric machine learning (ML) approaches, whose particular focus is demand forecasting in retail. At its core, this platform enables the training and application of probabilistic demand forecasting models, and provides convenient abstractions and support functionality for forecasting problems. The platform comprises of a complex end-to-end machine learning system built on Apache Spark, which includes data preprocessing, feature engineering, distributed learning, as well as evaluation, experimentation and ensembling. Furthermore, it meets the demands of a production system and scales to large catalogues containing millions of items. We describe the challenges of building such a platform and discuss our design decisions. We detail aspects on several levels of the system, such as a set of general distributed learning schemes, our machinery for ensembling predictions, and a high-level dataflow abstraction for modeling complex ML pipelines. To the best of our knowledge, we are not aware of prior work on real-world demand forecasting systems which rivals our approach in terms of scalability.Keywords
This publication has 15 references indexed in Scilit:
- Spark SQL2015
- Efficient sample generation for scalable meta learning2015
- Clustering to forecast sparse time-series data2015
- Materialization optimizations for feature selection workloads2014
- The Stratosphere platform for big data analyticsThe VLDB Journal, 2014
- Hybrid parallelization strategies for large-scale machine learning in SystemMLProceedings of the VLDB Endowment, 2014
- Large-scale machine learning at twitter2012
- Distributed GraphLabProceedings of the VLDB Endowment, 2012
- SCOPEProceedings of the VLDB Endowment, 2008
- A Limited Memory Algorithm for Bound Constrained OptimizationSIAM Journal on Scientific Computing, 1995