Tomer Geva
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Tomer Geva has a total of 12 co-authors. Here are the top co-authors by number of shared publications:
- Inbal Yahav
Coller School of Management, Tel-Aviv University, Tel Aviv-Yafo 6997801, Israel
2 shared publications - Maytal Saar-Tsechansky
University of Texas at Austin
2 shared publications - Anat Goldstein
Department of Industrial Engineering and Management, Ariel University, Ariel, Israel
1 shared publication Erik Brynjolfsson
1 shared publicationHarel Lustiger
1 shared publication- Harel Lustiger
Coller School of Management, Tel Aviv University
1 shared publication
Latest Publications
- 1 June 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Information Systems Research
- Vol. 37 (2) , 782-804
Quality Control for Crowd Workers and for Language Models: A Framework for Free-Text Response Evaluation with No Ground Truth
As businesses increasingly rely on large language models (LLMs) for tasks such as customer service and information retrieval, ensuring the accuracy of their responses is a critical challenge. Traditional verification is costly, slow, and often requires scarce domain experts. We introduce the automated quality evaluation based on textual responses (AQER) framework, a novel, cost-effective method to assess the correctness of free-text answers from both LLMs and human workers without needing preexisting correct answers. AQER works by intelligently aggregating multiple responses to the same question, leveraging the wisdom of the crowd to create a reliable synthetic correct answer, followed by an iterative procedure that accounts for response quality cues. AQER obtains state-of-the-art performance compared with existing automated response evaluation baselines. For managers AQER offers a scalable, data-driven method to (i) evaluate and select the best performing LLMs for specific organizational needs and use cases, (ii) continuously monitor artificial intelligence (AI) performance to ensure reliability and accountability across different model versions, and (iii) manage the quality of crowd workers essential for high-quality AI training and validation. AQER, thus, offers a robust mechanism for improving model performance and mitigating the significant financial and reputational risks associated with deploying untrustworthy generative AI technologies.
- 1 June 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Information Systems Research
- Vol. 37 (2) , 782-804
Quality Control for Crowd Workers and for Language Models: A Framework for Free-Text Response Evaluation with No Ground Truth
As businesses increasingly rely on large language models (LLMs) for tasks such as customer service and information retrieval, ensuring the accuracy of their responses is a critical challenge. Traditional verification is costly, slow, and often requires scarce domain experts. We introduce the automated quality evaluation based on textual responses (AQER) framework, a novel, cost-effective method to assess the correctness of free-text answers from both LLMs and human workers without needing preexisting correct answers. AQER works by intelligently aggregating multiple responses to the same question, leveraging the wisdom of the crowd to create a reliable synthetic correct answer, followed by an iterative procedure that accounts for response quality cues. AQER obtains state-of-the-art performance compared with existing automated response evaluation baselines. For managers AQER offers a scalable, data-driven method to (i) evaluate and select the best performing LLMs for specific organizational needs and use cases, (ii) continuously monitor artificial intelligence (AI) performance to ensure reliability and accountability across different model versions, and (iii) manage the quality of crowd workers essential for high-quality AI training and validation. AQER, thus, offers a robust mechanism for improving model performance and mitigating the significant financial and reputational risks associated with deploying untrustworthy generative AI technologies.
- 1 July 2025
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 71 (7) , 5696-5721
A Machine Learning Framework for Assessing Experts’ Decision Quality
Expert workers make non-trivial decisions with significant implications. Experts’ decision accuracy is, thus, a fundamental aspect of their judgment quality, key to both management and consumers of experts’ services. Yet, in many important settings, transparency in experts’ decision quality is rarely possible because ground truth data for evaluating the experts’ decisions is costly and available only for a limited set of decisions. Furthermore, different experts typically handle exclusive sets of decisions, and thus, prior solutions that rely on the aggregation of multiple experts’ decisions for the same instance are inapplicable. We first formulate the problem of estimating experts’ decision accuracy in this setting and then develop a machine–learning–based framework to address it. Our method effectively leverages both abundant historical data on workers’ past decisions and scarce decision instances with ground truth labels. Using both semi-synthetic data based on publicly available data sets and purposefully compiled data sets on real workers’ decisions, we conduct extensive empirical evaluations of our method’s performance relative to alternatives. The results show that our approach is superior to existing alternatives across diverse settings, including settings that involve different data domains, experts’ qualities, and amounts of ground truth data. To our knowledge, this paper is the first to posit and address the problem of estimating experts’ decision accuracies from historical data with scarce ground truth, and it is the first to offer comprehensive results for this problem setting, establishing the performances that can be achieved across settings as well as the state-of-the-art performance on which future work can build. This paper was accepted by Anindya Ghose, information systems. Funding: T. Geva acknowledges research grants from the Jeremy Coller Foundation and from the Henry Crown Institute for Business Research. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.03357 .
- 1 July 2025
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 71 (7) , 5696-5721
A Machine Learning Framework for Assessing Experts’ Decision Quality
Expert workers make non-trivial decisions with significant implications. Experts’ decision accuracy is, thus, a fundamental aspect of their judgment quality, key to both management and consumers of experts’ services. Yet, in many important settings, transparency in experts’ decision quality is rarely possible because ground truth data for evaluating the experts’ decisions is costly and available only for a limited set of decisions. Furthermore, different experts typically handle exclusive sets of decisions, and thus, prior solutions that rely on the aggregation of multiple experts’ decisions for the same instance are inapplicable. We first formulate the problem of estimating experts’ decision accuracy in this setting and then develop a machine–learning–based framework to address it. Our method effectively leverages both abundant historical data on workers’ past decisions and scarce decision instances with ground truth labels. Using both semi-synthetic data based on publicly available data sets and purposefully compiled data sets on real workers’ decisions, we conduct extensive empirical evaluations of our method’s performance relative to alternatives. The results show that our approach is superior to existing alternatives across diverse settings, including settings that involve different data domains, experts’ qualities, and amounts of ground truth data. To our knowledge, this paper is the first to posit and address the problem of estimating experts’ decision accuracies from historical data with scarce ground truth, and it is the first to offer comprehensive results for this problem setting, establishing the performances that can be achieved across settings as well as the state-of-the-art performance on which future work can build. This paper was accepted by Anindya Ghose, information systems. Funding: T. Geva acknowledges research grants from the Jeremy Coller Foundation and from the Henry Crown Institute for Business Research. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.03357 .
- 3 September 2020
- Published by SAGE Publications in Production and Operations Management
- Vol. 30 (1) , 127-144
Who Is a Better Decision Maker? Data‐Driven Expert Ranking Under Unobserved Quality
The capacity to rank expert workers by their decision quality is a key managerial task of substantial significance to business operations. However, when no ground truth information is available on experts’ decisions, the evaluation of expert workers typically requires enlisting peer‐experts, and this form of evaluation is prohibitively costly in many important settings. In this work, we develop a data‐driven approach for producing effective rankings based on the decision quality of expert workers; our approach leverages historical data on past decisions, which are commonly available in organizational information systems. Specifically, we first formulate a new business data science problem: Ranking Expert decision makers’ unobserved decision Quality (REQ) using only historical decision data and excluding evaluation by peer experts. The REQ problem is challenging because the correct decisions in our settings are unknown (unobserved) and because some of the information used by decision makers might not be available for retrospective evaluation. To address the REQ problem, we develop a machine‐learning–based approach and analytically and empirically explore conditions under which our approach is advantageous. Our empirical results over diverse settings and datasets show that our method yields robust performance: Its rankings of expert workers are consistently either superior or at least comparable to those obtained by the best alternative approach. Accordingly, our method constitutes a de facto benchmark for future research on the REQ problem.
- 3 September 2020
- Published by SAGE Publications in Production and Operations Management
- Vol. 30 (1) , 127-144
Who Is a Better Decision Maker? Data‐Driven Expert Ranking Under Unobserved Quality
The capacity to rank expert workers by their decision quality is a key managerial task of substantial significance to business operations. However, when no ground truth information is available on experts’ decisions, the evaluation of expert workers typically requires enlisting peer‐experts, and this form of evaluation is prohibitively costly in many important settings. In this work, we develop a data‐driven approach for producing effective rankings based on the decision quality of expert workers; our approach leverages historical data on past decisions, which are commonly available in organizational information systems. Specifically, we first formulate a new business data science problem: Ranking Expert decision makers’ unobserved decision Quality (REQ) using only historical decision data and excluding evaluation by peer experts. The REQ problem is challenging because the correct decisions in our settings are unknown (unobserved) and because some of the information used by decision makers might not be available for retrospective evaluation. To address the REQ problem, we develop a machine‐learning–based approach and analytically and empirically explore conditions under which our approach is advantageous. Our empirical results over diverse settings and datasets show that our method yields robust performance: Its rankings of expert workers are consistently either superior or at least comparable to those obtained by the best alternative approach. Accordingly, our method constitutes a de facto benchmark for future research on the REQ problem.
- 25 November 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 33 (6) , 2380-2391
Data-Driven Link Screening for Increasing Network Predictability
Prediction methods applied to digital network data offer powerful capabilities that have radically affected a host of industries and services. Specifically, numerous works have shown that the use of network information to predict focal node properties produces significantly more accurate results compared with exclusive reliance on other types of data. In this study, we propose that it may be possible to improve network-information-based predictions by identifying network links that actually carry predictive power for a given prediction task. For this purpose, we suggest a problem referred to as the problem of Increasing Network Predictability (INP) by data-driven link screening. To address this problem we develop a new algorithm with three different implementations. We find that the algorithm is robust and consistently outperforms baseline link selection methods. We thus suggest that our algorithm has the potential to improve the efficacy of network data use for classification purposes.
- 25 November 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 33 (6) , 2380-2391
Data-Driven Link Screening for Increasing Network Predictability
Prediction methods applied to digital network data offer powerful capabilities that have radically affected a host of industries and services. Specifically, numerous works have shown that the use of network information to predict focal node properties produces significantly more accurate results compared with exclusive reliance on other types of data. In this study, we propose that it may be possible to improve network-information-based predictions by identifying network links that actually carry predictive power for a given prediction task. For this purpose, we suggest a problem referred to as the problem of Increasing Network Predictability (INP) by data-driven link screening. To address this problem we develop a new algorithm with three different implementations. We find that the algorithm is robust and consistently outperforms baseline link selection methods. We thus suggest that our algorithm has the potential to improve the efficacy of network data use for classification purposes.
- Open Access
- 13 June 2019
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 33 (6) , 1625-1673
More for less: adaptive labeling payments in online labor markets
- 13 June 2019
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 33 (6) , 1625-1673
Open AccessMore for less: adaptive labeling payments in online labor markets
- Open Access
- 1 January 2016
- Published by Elsevier in SSRN Electronic Journal
More for Less: Adaptive Labeling Payment for Online Labor Markets
Predictive modeling has emerged as integral to the efficient operations and competitive strategies of firms across industries. Because many important predictive tasks require human intelligence to label training instances, online crowdsourcing markets have become a promising platform for large-scale labeling. However, prior research found major quality issues in such markets. In particular, very different tradeoffs arise between payment offered to labelers and the quality of labeling under different settings, and, more broadly, work quality may change over time and with changes in the competitive market settings. Further, determining the effect of labeling quality on the expected improvement in predictive performance is also challenging. Therefore, effective means for dealing with these challenges are essential for a growing reliance on these markets for predictive modelling. In this paper, we propose the new data science problem of Adaptive Labeling Payment (ALP): how to determine and continuously adapt the payment offered to crowd workers, before they undertake a labeling task, so as to produce a given predictive performance cost-effectively. We develop an ALP framework and derive a novel ALP method, which we evaluate extensively over a wide variety of market conditions. We find that our ALP method yields substantial cost savings and robust performance that can be relied on by businesses over a wide variety of settings.
- 1 January 2016
- Published by Elsevier in SSRN Electronic Journal
Open AccessMore for Less: Adaptive Labeling Payment for Online Labor Markets
Predictive modeling has emerged as integral to the efficient operations and competitive strategies of firms across industries. Because many important predictive tasks require human intelligence to label training instances, online crowdsourcing markets have become a promising platform for large-scale labeling. However, prior research found major quality issues in such markets. In particular, very different tradeoffs arise between payment offered to labelers and the quality of labeling under different settings, and, more broadly, work quality may change over time and with changes in the competitive market settings. Further, determining the effect of labeling quality on the expected improvement in predictive performance is also challenging. Therefore, effective means for dealing with these challenges are essential for a growing reliance on these markets for predictive modelling. In this paper, we propose the new data science problem of Adaptive Labeling Payment (ALP): how to determine and continuously adapt the payment offered to crowd workers, before they undertake a labeling task, so as to produce a given predictive performance cost-effectively. We develop an ALP framework and derive a novel ALP method, which we evaluate extensively over a wide variety of market conditions. We find that our ALP method yields substantial cost savings and robust performance that can be relied on by businesses over a wide variety of settings.
- Open Access
- 1 January 2014
- Published by Elsevier in SSRN Electronic Journal
Crowd-Squared: Amplifying the Predictive Power of Large-Scale Crowd-Based Data
Large-scale data generated by crowds provide a myriad of opportunities for monitoring and modeling people's intentions, preferences, and opinions. A crucial ste
- 1 January 2014
- Published by Elsevier in SSRN Electronic Journal
Open AccessCrowd-Squared: Amplifying the Predictive Power of Large-Scale Crowd-Based Data
Large-scale data generated by crowds provide a myriad of opportunities for monitoring and modeling people's intentions, preferences, and opinions. A crucial ste