"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
Top Cited Papers
Open Access
- 17 April 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Human-Computer Interaction
- Vol. 8 (CSCW1) , 1-34
Abstract
Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationalization of machine learning, or MLOps, requires MLEs to have proficiency in data science and engineering. When considered holistically, the job seems staggering---how do MLEs do MLOps, and what are their unaddressed challenges? To address these questions, we conducted semi-structured ethnographic interviews with 18 MLEs working on various applications, including chatbots, autonomous vehicles, and finance. We find that MLEs engage in a workflow of (i) data preparation, (ii) experimentation, (iii) evaluation throughout a multi-staged deployment, and (iv) continual monitoring and response. Throughout this workflow, MLEs collaborate extensively with data scientists, product stakeholders, and one another, supplementing routine verbal exchanges with communication tools ranging from Slack to organization-wide ticketing and reporting systems. We introduce the 3Vs of MLOps: velocity, visibility, and versioning --- three virtues of successful ML deployments that MLEs learn to balance and grow as they mature. Finally, we discuss design implications and opportunities for future work.Keywords
All Related Versions
Funding Information
- National Science Foundation (DGE2243822)
This publication has 72 references indexed in Scilit:
- MLOps: A Taxonomy and a MethodologyIEEE Access, 2022
- MLOps - Definitions, Tools and Challenges2022
- On Continuous Integration / Continuous Delivery for Automated Deployment of Machine Learning Models using MLOps2021
- Why AI/Data Science Projects Fail2021
- DevOps in practice: A multiple case study of five companiesInformation and Software Technology, 2019
- What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study2018
- Snakemake—a scalable bioinformatics workflow engineBioinformatics, 2012
- A unifying view on dataset shift in classificationPattern Recognition, 2012
- Importance-Weighted Cross-Validation for Covariate Shift2006
- An Introduction to Agile Methods2004