HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale
Open Access
- 30 April 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3837-3848
Abstract
Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts’ ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to evaluate a large number of design ideas in a short amount of time while being cost-effective. Workers’ evaluation is, however, less reliable and might substantially differ from experts’ evaluation. In this work, we investigate workers’ rating behavior and compare it with experts. First, we instrument a crowdsourcing study where we asked workers to evaluate design ideas from three innovation challenges. We show that workers share similar insights with experts but tend to rate more generously and weigh certain criteria more importantly. Next, we develop a hybrid human-AI approach that combines a machine learning model with crowdsourcing to evaluate ideas. Our approach models workers’ reliability and bias while leveraging ideas’ textual content to train a machine learning model. It is able to incorporate experts’ ratings whenever available, to supervise the model training and infer worker performance. Results show that our framework outperforms baseline methods and requires significantly less training data from experts, thus providing a viable solution for evaluating ideas at scale.Keywords
This publication has 37 references indexed in Scilit:
- "Why Should I Trust You?"2016
- Crowdsourcing ideas: Involving ordinary users in the ideation phase of new product developmentResearch Policy, 2016
- High-speed idea filtering with the bag of lemonsDecision Support Systems, 2015
- Fostering Customer Ideation in Crowdsourcing Community: The Role of Peer-to-peer and Peer-to-firm InteractionsJournal of Interactive Marketing, 2015
- InspirationWall2015
- When Crowdsourcing Fails: A Study of Expertise on Crowdsourced Design EvaluationJournal of Mechanical Design, 2015
- Refinements to the variety metric for idea evaluationDesign Studies, 2013
- The Extended Value of Design: An Advanced Design PerspectiveDesign Management Journal, 2011
- Refined metrics for measuring ideation effectivenessDesign Studies, 2009
- The variational approximation for Bayesian inferenceIEEE Signal Processing Magazine, 2008