A Machine Learning Framework for Assessing Experts’ Decision Quality
- 1 July 2025
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 71 (7) , 5696-5721
Abstract
Expert workers make non-trivial decisions with significant implications. Experts’ decision accuracy is, thus, a fundamental aspect of their judgment quality, key to both management and consumers of experts’ services. Yet, in many important settings, transparency in experts’ decision quality is rarely possible because ground truth data for evaluating the experts’ decisions is costly and available only for a limited set of decisions. Furthermore, different experts typically handle exclusive sets of decisions, and thus, prior solutions that rely on the aggregation of multiple experts’ decisions for the same instance are inapplicable. We first formulate the problem of estimating experts’ decision accuracy in this setting and then develop a machine–learning–based framework to address it. Our method effectively leverages both abundant historical data on workers’ past decisions and scarce decision instances with ground truth labels. Using both semi-synthetic data based on publicly available data sets and purposefully compiled data sets on real workers’ decisions, we conduct extensive empirical evaluations of our method’s performance relative to alternatives. The results show that our approach is superior to existing alternatives across diverse settings, including settings that involve different data domains, experts’ qualities, and amounts of ground truth data. To our knowledge, this paper is the first to posit and address the problem of estimating experts’ decision accuracies from historical data with scarce ground truth, and it is the first to offer comprehensive results for this problem setting, establishing the performances that can be achieved across settings as well as the state-of-the-art performance on which future work can build. This paper was accepted by Anindya Ghose, information systems. Funding: T. Geva acknowledges research grants from the Jeremy Coller Foundation and from the Henry Crown Institute for Business Research. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.03357 .Keywords
All Related Versions
This publication has 52 references indexed in Scilit:
- More for less: adaptive labeling payments in online labor marketsData Mining and Knowledge Discovery, 2019
- Time of Day and the Decision to Prescribe AntibioticsJAMA Internal Medicine, 2014
- Repeated labeling using multiple noisy labelersData Mining and Knowledge Discovery, 2013
- Thinking, feeling and deciding: The influence of emotions on the decision making and performance of tradersJournal of Organizational Behavior, 2011
- When emotions get the better of us: the effect of contextual top-down processing on matching fingerprintsApplied Cognitive Psychology, 2005
- Counting Deaths Due to Medical ErrorsJAMA, 2002
- Expert decision makingSystems Engineering, 1999
- A model of the effects of audit task complexityAccounting, Organizations and Society, 1994
- Competence in experts: The role of task characteristicsOrganizational Behavior and Human Decision Processes, 1992
- Availability: A heuristic for judging frequency and probabilityCognitive Psychology, 1973