Idea Evaluation for Solutions to Specialized Problems: Leveraging the Potential of Crowds and Large Language Models
Open Access
- 28 June 2025
- journal article
- research article
- Published by Springer Nature in Group Decision and Negotiation
- Vol. 34 (4) , 903-932
Abstract
Complex problems such as climate change pose severe challenges to societies worldwide. To overcome these challenges, digital innovation contests have emerged as a promising tool for idea generation. However, assessing idea quality in innovation contests is becoming increasingly problematic in domains where specialized knowledge is needed. Traditionally, expert juries are responsible for idea evaluation in such contests. However, experts are a substantial bottleneck as they are often scarce and expensive. To assess whether expert juries could be replaced, we consider two approaches. We leverage crowdsourcing and a Large Language Model (LLM) to evaluate ideas, two approaches that are similar in terms of the aggregation of collective knowledge and could therefore be close to expert knowledge. We compare expert jury evaluations from innovation contests on climate change with crowdsourced and LLM’s evaluations and assess performance differences. Results indicate that crowds and LLMs have the ability to evaluate ideas in the complex problem domain while contest specialization—the degree to which a contest relates to a knowledge-intensive domain rather than a broad field of interest—is an inhibitor of crowd evaluation performance but does not influence the evaluation performance of LLMs. Our contribution lies with demonstrating that crowds and LLMs (as opposed to traditional expert juries) are suitable for idea evaluation and allows innovation contest operators to integrate the knowledge of crowds and LLMs to reduce the resource bottleneck of expert juries.Keywords
Funding Information
- Bundesministerium für Forschung, Technologie und Raumfahrt (16DHBKI002, 16DHBKI003)
- Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg (16DHBKI002, 16DHBKI003)
- Hessisches Ministerium für Wissenschaft und Kunst (16DHBKI002, 16DHBKI003)
- Universität Hohenheim
This publication has 59 references indexed in Scilit:
- The design of human-artificial intelligence systems in decision sciences: A look back and directions forwardDecision Support Systems, 2024
- Generative AIBusiness & Information Systems Engineering, 2023
- On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practiceInternational Journal of Research in Marketing, 2023
- BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in CrowdsourcingGroup Decision and Negotiation, 2022
- Facilitating like Darwin: Supporting cross-fertilisation in crowdsourcingDecision Support Systems, 2020
- Idea evaluation in innovation contest platforms: A network perspectiveDecision Support Systems, 2018
- Rethinking the Boundaries of Cognitive Load Theory in Complex LearningEducational Psychology Review, 2015
- High-speed idea filtering with the bag of lemonsDecision Support Systems, 2015
- A comparison of ranking and rating procedures for value system measurementEuropean Journal of Social Psychology, 1980
- Dilemmas in a general theory of planningPolicy Sciences, 1973