An Improved Best‐Worst Method Integrated With Combined Compromise Solution for Evaluating Large Language Models
Open Access
- 1 January 2025
- journal article
- Published by Wiley in International Journal of Intelligent Systems
- Vol. 2025 (1)
Abstract
The emergence of large language models (LLMs) has substantially changed the artificial intelligence field, enabling its wide use over different domains. As various LLM alternatives have been developed, the current study proposes a novel decision‐support framework for evaluating and benchmarking LLMs based on multicriteria decision‐making (MCDM) techniques. In the proposed framework, an improved version of the best‐worst method (BWM) is proposed to effectively reduce the computational complexity of assigning a critical weight for the evaluation criteria of LLMs. Then, the improved BWM is integrated with the combined compromise solution (CoCoSo) method for ranking LLM alternatives. Findings show that the improved BWM successfully computes the criteria weights with low computational complexity compared to the original BWM. According to the enhanced BWM, the ‘factual errors’ criterion received the highest significant weight (0.2681), while the ‘logical inconsistencies’ criteria obtained the lowest (0.0827). The rest of the criteria were distributed in between that range. Subsequently, CoCoSo ranked the involved LLM alternatives in two different runs based on the extracted weights. Sensitivity analysis was employed to evaluate the effect of the assessment criteria on LLMs’ evaluation.Keywords
This publication has 60 references indexed in Scilit:
- A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept ExtractionJournal of Medical Systems, 2024
- Welcome to the Era of ChatGPT et al.Business & Information Systems Engineering, 2023
- Based on neutrosophic fuzzy environment: a new development of FWZIC and FDOSM for benchmarking smart e-tourism applicationsComplex & Intelligent Systems, 2022
- Green supplier selection for textile industry: a case study using BWM-TODIM integration under interval type-2 fuzzy setsEnvironmental Science and Pollution Research, 2021
- Dempster–Shafer theory for classification and hybridised models of multi-criteria decision analysis for prioritisation: a telemedicine framework for patients with heart diseasesJournal of Ambient Intelligence and Humanized Computing, 2021
- New mHealth hospital selection framework supporting decentralised telemedicine architecture for outpatient cardiovascular disease-based integrated techniques: Haversine-GPS and AHP-VIKORJournal of Ambient Intelligence and Humanized Computing, 2021
- A systematic review into the assessment of medical apps: motivations, challenges, recommendations and methodological aspectHealth and Technology, 2020
- Multi-criteria decision making in Pythagorean fuzzy environmentApplied Intelligence, 2019
- Multiclass Benchmarking Framework for Automated Acute Leukaemia Detection and Classification Based on BWM and Group-VIKORJournal of Medical Systems, 2019
- The complexity of approximating a nonlinear programMathematical Programming, 1995