Evaluating an LLM Pipeline for End-to-End MCDA
Open Access
- 15 October 2026
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 16 (20) , 10210
Abstract
Multi-Criteria Decision Analysis (MCDA) requires human involvement at every stage of its process. This paper proposes a systematic LLM evaluation protocol for end-to-end MCDA pipelines, using published expert rankings as a benchmark. In addition, it implements a four-stage LLM pipeline as a case study applying the protocol to three leading LLMs across five benchmark problems selected to vary in application domain and in the number of alternatives. The protocol comprises 225 independent pipeline executions (3 models × 5 repetitions × 3 temperatures × 5 problems); each execution applies three MCDA methods (LLM-weighted linear scoring, TOPSIS, PROMETHEE II) to the same LLM-generated score matrix, yielding 675 method-specific outputs in total. Consistent with a smaller pilot study of this protocol, differences between models are small relative to problem identity (Kruskal–Wallis, H = 1.10, p = 0.577, η2 = 0.005; a small but significant effect emerges once problem identity is controlled for, η2 = 0.6%), while problem identity remains the dominant performance factor—and a larger one than previously observed (η2 = 0.761, H = 170.51, p < 0.001, N = 225 executions). Kendall’s τ ranges from −0.34 on a ten-alternative problem to 0.56 on a four-alternative problem; a permutation test against random rankings indicates that two of the five benchmark problems produced agreement below chance level, while two problems produced agreement above chance level; since the test treats executions as independent, these deviations should be read as descriptive rather than inferential. Top-3 Accuracy ranges from 10.1% to 72.1% across problems, relative to chance baselines of 30% to 75%. Reliability varies strongly across problems, and the compensatory/non-compensatory distinction from the pilot does not fully explain this. In this pipeline, the scoring stage receives the names of the alternatives and criteria but not the numerical decision data, so the results reflect the LLM’s prior knowledge; its role as an explanatory factor is proposed as a hypothesis for future testing.Keywords
This publication has 22 references indexed in Scilit:
- Large Language Models in MCDA Weight Elicitation: A Systematic Review2026
- Large language models present new questions for decision supportInternational Journal of Information Management, 2024
- Multicriteria decision making and goal programming for determination of electric automobile aimed at sustainable green environment: a case studyEnvironment Systems and Decisions, 2022
- Modeling forest fire risk based on GIS-based analytical hierarchy process and statistical analysis in Mediterranean regionEcological Informatics, 2022
- Aircraft selection by applying AHP and TOPSIS in interval type-2 fuzzy setsJournal of Air Transport Management, 2020
- Application of Fuzzy Reference Ideal Method (FRIM) to the military advanced training aircraft selectionApplied Soft Computing, 2020
- Cloud plan selection under requirements of multiple applicationsSecurity and Privacy, 2018
- Multiple Criteria Decision Analysis2016
- Analytic Hierarchy Process2005
- Multiple Criteria Decision Analysis2002