A taxonomy of reliability indices for cluster‐level constructs measured by multiple individual‐level indicators
Open Access
- 14 October 2026
- journal article
- research article
- Published by Wiley in British Journal of Mathematical and Statistical Psychology
- Vol. 79 (3) , 513-546
Abstract
Generalizability theory (GT) provides a variety of intraclass correlation coefficients (ICCs) to quantify reliability across different conditions of measurement—for example, scale and interrater reliability quantify generalization across items and raters, respectively. In a multilevel modelling (MLM) context, an ICC is often used to quantify the proportion of a subject‐level variable's variance that is attributable to cluster‐level differences (e.g., between schools or neighbourhoods). When data from cluster members are their ratings of a group‐level characteristic, the ICC is an inter‐rater reliability coefficient, interpreted as the degree to which observations of a cluster generalize across members of that cluster. Structural equation modelling (SEM) is becoming a popular framework for modelling GT, and the ‘omega’ indices of composite reliability—which are functions of SEM parameters—are special cases of GT (i.e., how measurements of a construct generalize across indicators). The merger of MLM and SEM (i.e., ML‐SEM) provides a unique opportunity to explore how well measurements of cluster‐level constructs generalize across multiple measurement conditions (indicators, cluster members, or both). A taxonomy of ICCs is explicated to appropriately quantify reliability across aspects of a procedure for measuring cluster‐level constructs, adding value to the limited ICCs currently considered in ML‐SEM applications.Keywords
Funding Information
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (VI.Vidi.201.009)
This publication has 48 references indexed in Scilit:
- Applications of generalizability theory and their relations to classical test theory and structural equation modeling.Psychological Methods, 2018
- Cronbach's alpha reliability: Interval estimation, hypothesis testing, and sample size planningJournal of Organizational Behavior, 2014
- Reliability estimation in a multilevel confirmatory factor analysis framework.Psychological Methods, 2014
- Linear and Generalized Linear Mixed Models and Their Applications2007
- Generalizability Theory2001
- A Unifying Expression for the Maximal Reliability of a Linear CompositePsychometrika, 1997
- Structural Equations with Latent Variables1989
- Statistical Analysis of Sets of Congeneric TestsPsychometrika, 1971
- Alpha-Maximized Factor Analysis (Alphamax): Its Relation to Alpha and Canonical Factor AnalysisPsychometrika, 1968
- Coefficient Alpha and the Internal Structure of TestsPsychometrika, 1951