Leveraging Order-Theoretic Tournament Graphs for Assessing Internal Consistency in Survey-Based Instruments Across Diverse Scenarios
Authors:
- Muhammad Umair DanishMDMuhammad Umair Danish
- Umair RehmanURUmair Rehman
- Katarina GrolingerKatarina Grolinger
Preprint
Abstract
This paper introduces Monotone Delta, an order-theoretic measure designed to enhance the reliability assessment of survey-based instruments in human-machine interactions. Traditional reliability measures, such as Cronbach's Alpha and McDonald's Omega, often yield misleading estimates due to their sensitivity to redundancy, multidimensional constructs, and assumptions of normality and uncorrelated errors. These limitations can compromise decision-making in human-centric evaluations, where survey instruments inform adaptive interfaces, cognitive workload assessments, and human-AI trust models. Monotone Delta addresses these issues by quantifying internal consistency through the minimization of ordinal contradictions and alignment with a unidimensional latent order using weighted tournaments. Unlike traditional approaches, it operates without parametric or model-based assumptions. We conducted theoretical analyses and experimental evaluations on four challenging scenarios: tau-equivalence, redundancy, multidimensionality, and non-normal distributions, and proved that Monotone Delta provides more stable reliability assessments compared to existing methods. The Monotone Delta is a valuable alternative for evaluating questionnaire-based assessments in psychology, human factors, healthcare, and interactive system design, enabling organizations to optimize survey instruments, reduce costly redundancies, and enhance confidence in human-system interactions.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Reliability and omega hierarchical in multidimensional data: A comparison of various estimators.Psychological Methods, 2025
- Comparing attentional disengagement between Prolific and MTurk samplesScientific Reports, 2023
- Neither Cronbach’s Alpha nor McDonald’s Omega: A Commentary on Sijtsma and PfadtPsychometrika, 2021
- Clarity on Cronbach’s Alpha UseJournal of Chemical Education, 2020
- Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…Communication Methods and Measures, 2020
- Conducting Measurement Invariance Tests with Ordinal Data: A Guide for Social Work ResearchersJournal of the Society for Social Work and Research, 2015
- Cronbach's Alpha: Review of Limitations and Associated RecommendationsJournal of Psychology in Africa, 2010
- The Greatest Lower Bound to the Reliability of a Test and the Hypothesis of UnidimensionalityPsychometrika, 2004
- R&D/marketing communication during the fuzzy front-endIEEE Transactions on Engineering Management, 1995
- Minimum Feedback Arc Sets for a Directed GraphIEEE Transactions on Circuit Theory, 1963