Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
Authors:
- Samantha RobertsonSRSamantha Robertson
- Zijie J. WangZWZijie J. Wang
- Dominik MoritzDMDominik Moritz
- Mary Beth KeryMKMary Beth Kery
- Fred HohmanFHFred Hohman
Preprint
Abstract
Machine learning (ML) models can fail in unexpected ways in the real world, but not all model failures are equal. With finite time and resources, ML practitioners are forced to prioritize their model debugging and improvement efforts. Through interviews with 13 ML practitioners at Apple, we found that practitioners construct small targeted test sets to estimate an error's nature, scope, and impact on users. We built on this insight in a case study with machine translation models, and developed Angler, an interactive visual analytics tool to help practitioners prioritize model improvements. In a user study with 7 machine translation experts, we used Angler to understand prioritization practices when the input space is infinite, and obtaining reliable signals of model quality is expensive. Our study revealed that participants could form more interesting and user-focused hypotheses for prioritization by analyzing quantitative summary statistics and qualitatively assessing data by reading sentences.Keywords
All Related Versions
- Version 1, 2023-04-12, ArXiv
- Published version: , 1.
This publication has 89 references indexed in Scilit:
- Visualization-based improvement of neural machine translationComputers & Graphics, 2022
- On Continuous Integration / Continuous Delivery for Automated Deployment of Machine Learning Models using MLOps2021
- Application of Uniform Manifold Approximation and Projection (UMAP) in spectral imaging of artworksSpectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy, 2021
- A Pragmatic Assessment of Google Translate for Emergency Department InstructionsJournal of General Internal Medicine, 2021
- Racial disparities in automated speech recognitionProceedings of the National Academy of Sciences, 2020
- Assessing the Use of Google Translate for Spanish and Chinese Translations of Emergency Department Discharge InstructionsJAMA Internal Medicine, 2019
- Multivariate Density Estimation2015
- iBLEU: Interactively Debugging and Scoring Statistical Machine Translation Systems2011
- Evaluating Evaluation Methods for Generation in the Presence of Variation2005
- LIII. On lines and planes of closest fit to systems of points in spaceJournal of Computers in Education, 1901