Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training
Authors:
Preprint
Abstract
Domain-specific accelerators (DSAs) are integrated in datacenter-scale clusters across industry to train increasingly-complex deep learning models over massive datasets. As innovations in DSAs continue to increase training efficiency and throughput, the data storage and ingestion (DSI) pipeline, the systems and hardware responsible for storing and preprocessing training data, will dominate and constrain training capacity. Similar innovation in DSI is urgent, demanding an in-depth understanding of DSI systems, infrastructure, and characteristics. To this end, this paper presents Meta's end-to-end DSI pipeline, composed of a central data warehouse built on distributed storage and a Data PreProcessing Service (DPP) that scales to eliminate data stalls. We characterize how hundreds of models are collaboratively trained across our global fleet, how massive and evolving datasets are stored and read, and how online preprocessing places intense demands on our underlying hardware. We synthesize key takeaways from our characterization and close with a discussion of lessons learned and research opportunities for both industry and academia.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Understanding Training Efficiency of Deep Learning Recommendation Models at Scale2021
- Accelerating Data Loading in Deep Neural Network Training2019
- AutoAugment: Learning Augmentation Strategies From Data2019
- The Snowflake Elastic Data Warehouse2016
- Profiling a warehouse-scale computer2015
- Practical Lessons from Predicting Clicks on Ads at Facebook2014
- Microsoft COCO: Common Objects in Context2014
- Naiad2013
- Discretized streams2013
- Leakage in data miningACM Transactions on Knowledge Discovery From Data, 2012