Erica Fu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Erica Fu has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Adam Perer
Carnegie Mellon University, USA
1 shared publication - Ameet Talwalkar
Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
1 shared publication - Donald Bertucci
School of Electrical Engineering and Computer Science, Oregon State University, United States
1 shared publication - Jason I. Hong
Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
1 shared publication - Kenneth Holstein
Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
1 shared publication - Ángel Alexander Cabrera
Carnegie Mellon University, United States
1 shared publication
Latest Publications
- Open Access
- 19 April 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-14
Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.
- 19 April 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-14
Open AccessZeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.