Which to Trust: Extracting Collective Wisdom Based on Opinion Quality Rank Learning
- 2 December 2025
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in INFORMS Journal on Computing
- Vol. 38 (5) , 1673-1692
Abstract
Aggregating diverse human opinions in the digital era is essential in harnessing collective wisdom to unravel intricate management dilemmas. The aggregation process has significant hurdles to overcome due to the heterogeneity in the quality of individual opinions. This study introduces a CrowdRank machine learning architecture to provide an innovative solution to the central problem of opinion aggregation by learning to rank individual opinions. CrowdRank operates through two phases. (1) It leverages a BNN, pretrained on historical data, to conduct pairwise opinion comparisons. This network, designed to capture meaningful interactions between opinion features, adheres to key axiomatic principles to ensure a principled evaluation of opinion quality. (2) CrowdRank employs expectation propagation to synthesize these microassessments into a coherent global ranking of individual opinions. We validated the efficacy of our approach through a large-scale empirical investigation using real-world financial analyst forecasts. The validation results demonstrated the superiority of our approach over existing methods in accurately predicting both pairwise and aggregate opinion rankings. Importantly, CrowdRank significantly improves the objectivity and precision of collective financial analyst forecasts. This study contributes a theoretically robust and practically validated innovation to opinion aggregation and charts a new path in the application of machine learning to enhance the synthesis of collective wisdom. History: Accepted by Ram Ramesh, Area Editor for Data Science and Machine Learning. Funding: This work was partly supported by the National Natural Science Foundation of China [Grants 92370204 and 71974031]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0835 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2024.0835 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .Keywords
This publication has 28 references indexed in Scilit:
- Decision aggregation with reliability propagationDecision Support Systems, 2023
- Identifying predictors of analyst rating quality: An ensemble feature selection approachInternational Journal of Forecasting, 2022
- Learning From Noisy Labels With Deep Neural Networks: A SurveyIEEE Transactions on Neural Networks and Learning Systems, 2022
- Management forecast errors and corporate investment efficiencyJournal of Contemporary Accounting & Economics, 2020
- Pairwise Learning to Rank by Neural Networks Revisited: Reconstruction, Theoretical Analysis and Practical Performance2020
- A physical model for efficient ranking in networksScience Advances, 2018
- Interlock Concentration and Analyst Forecast Accuracy: Value Implications of InterlockAustralian Accounting Review, 2018
- Analysts' experience and interpretation of discretionary accruals in predicting future earningsAdvances in Accounting, 2017
- Measure for Measure: The Relation between Forecast Accuracy and Recommendation Profitability of AnalystsJournal of Accounting Research, 2007
- Ranking from unbalanced paired-comparison dataBiometrika, 1987