Francis Tuerlinckx

Intraclass correlation coefficients (ICCs) quantify the reliability of measurements made on the same persons by different raters. In interrater reliability studies, consistency ICCs assess whether scores preserve the relative rankings of persons, while agreement ICCs evaluate the closeness of absolute scores across raters, thereby penalising systematic rater differences. In two-way mixed-effects designs, where raters are treated as fixed, conventional agreement ICCs raise a conceptual paradox: fixed-rater dispersion contributes to a source of disagreement despite the rater effects being constants, and it remains in the coefficient for all-rater averages even though those effects cancel in the average. In this paper, we resolve this paradox by introducing a finite-population sampling framework grounded in the correlation approach with hypothetical replications. Within this framework, we derive correlation-based agreement ICCs for both single ratings and all-rater averages, establishing a coherent probabilistic foundation that resolves the paradox. By considering various rater-sampling schemes, a family of agreement ICCs in mixed-effects designs is presented. We further show that the conventional agreement formulas can be recovered under independent sampling with replacement both within and between replicates. Thus, choosing an appropriate agreement ICC requires specifying how raters are sampled within and across replicates, rather than relying on the design alone.

Psychological processes are characterized by constant change and complex underlying dynamics and are often researched using intensive longitudinal data (ILD). Although such data are often analyzed with parametric models, these models impose strong and simplified assumptions about the system’s dynamic rules that can be unrealistic and may fail to capture phenomena such as multistability. Recent work has enabled estimation of the stability landscape of psychological systems from ILD, yet drawing statistical inferences from such landscapes remains challenging. In this work, we present a new method for statistical inference based on stability landscapes. The core of the method is bootstrapping, complemented by specialized techniques applied before and after the bootstrap procedure to make computation feasible and to effectively summarize information across bootstrap samples. Across three simulation studies with different data-generating mechanisms, we evaluated the method and, in particular, four approaches for summarizing information about the locations and depths of local minima. Overall, we found that using fixed reference points, such as the local minima from the original sample, helps guide the assignment of bootstrapped minima. When the data-generating mechanism differs from the estimated stability landscape, directional bias may occur; however, the relative locations of minima can still be retrieved. The method generally shows higher precision, whereas achieving high recall may require clear separation between minima, longer time series, and lower measurement noise. We also provide an empirical illustration using an existing dataset. An implementation is available in the latest version of the R package fitlandr.

Psychometric models use latent variables to quantify psychological traits, but statistical and substantive inferences based on these variables may depend on how the latent scale is parametrized. Consequently, the same fitted model can yield qualitatively different conclusions under different parametrizations. Building on Ramsay’s (1996, Behaviormetrika , 23, 3–16) geometric perspective, we reinterpret this problem using differential geometry. While Ramsay defined ability via an arc length in Euclidean space, psychometric models can be seen instead as a curved statistical manifold with the Fisher–Rao metric. This allows for a parametrization-invariant definition of ability, which we term geodesic ability. We illustrate this concept for unidimensional item response theory models, including the Rasch and 3PL models, and compare it to Ramsay’s arc-length approach.

Intensive longitudinal data (ILD) are increasingly used to study affect dynamics such as intraindividual variability and inertia. These parameters are often interpreted as substantive individual differences, yet they may be biased by stable response style tendencies, in particiular extreme response style (ERS). In the present study, we investigated whether ERS is systematically associated with affect dynamics estimated from multiple experience sampling method (ESM) datasets (total N = 1,254). Using a joint Bayesian IRTree–location–scale modeling approach, we estimated person-specific ERS alongside affective variability and inertia across multiple ESM datasets and synthesized study-level associations using a meta-analytic approach. We find that higher ERS is consistently associated with greater intraindividual variability. In contrast, we find no evidence that ERS was related to affective or emotional inertia. These results suggest that commonly used measures of intraindividual variability may partly reflect systematic response behavior rather than solely genuine affective fluctuations. Ignoring ERS may therefore bias variability estimates obtained from location–scale and related multilevel dynamic models. Overall, this study highlights the importance of accounting for response styles when modeling affect dynamics and interpreting person-specific variability parameters in ILD research.

With the advancement of technology, intensive longitudinal data (ILD) are increasingly collected to study dynamic, moment-to-moment psychological and physiological processes. Despite their widespread use, the issues of measurement error and reliability are often inadequately addressed, particularly in the presence of serial correlation and complex multilevel structures. This article introduces a three-level linear mixed-effects model that explicitly accounts for nested structures (moments within days within persons) and incorporates two distinct autoregressive processes for moment- and day-level serial correlations, partitioning measurement error from within-person variability. Building on this model, we delineate two approaches to defining reliability: a variance partitioning approach and a correlation approach. This framework yields a multifaceted set of reliability coefficients, enabling researchers to distinguish between-person from within-person reliability, and to assess measurement agreement across different time scales (e.g., moment-to-moment and day-to-day). We further discuss the relationship between the proposed reliability measures and those derived from simpler models. We apply our model to study the reliability of a single negative affect item in an ILD study in which students were measured multiple times a day for several days before and after receiving their exam results.

Psychological tasks are increasingly used to develop behavioral biomarkers for precision psychiatry, yet poor test-retest reliability of many task-derived summary measures has raised doubts about their use for stratifying patients, predicting treatment response or indexing clinically relevant mechanisms. In simulations of a fear-conditioning task, we establish two results. First, common summaries can mislead in two ways: they can appear unreliable because they are shaped by experimental context, such as trial order, or reliable because they capture stable but unintended features of responding rather than the target mechanism. Second, computational-model parameters can support stronger reliability claims when they more closely track the intended process, provided the model captures the underlying mechanism and the design makes it recoverable. Together, these findings show that reliability can systematically mislead mechanistic inference unless interpreted with respect to a generative model, the diagnosticity of the task design and assumptions linking the metric to the intended mechanism.

The intraclass correlation coefficient (ICC) is widely used to assess reliability, agreement, and consistency for quantitative measurements. Despite being termed a ``correlation'', its definition and interpretation can be ambiguous because the ICC is commonly computed using variance components, a procedure that may obscure its conceptual foundations as a measure of correlation. In this tutorial paper, we aim to clarify the conceptual foundations of the ICC and its relationship to correlation by establishing a consistent way of conceptualizing ICCs through simulations. We illustrate various forms of ICCs under the one-way nested, two-way crossed, and three-level fully nested random-effects models, and show how they can be interpreted as correlations between two hypothetical replicates of the measurement process. To make this concept intuitive, we introduce a visual grammar that maps out the data-generating mechanisms. We then use Monte Carlo simulations to empirically verify that this correlation approach perfectly converges with variance-based ICC formulas. Ultimately, this tutorial provides researchers with an intuitive conceptual compass to seamlessly understand, select, and interpret appropriate ICCs across diverse experimental designs.

With the advancement of technology, measures are more and more often collected in real-time, resulting in intensive longitudinal data (ILD). ILD allow us to study dynamic properties of one or more variables, such as moment-to-moment fluctuations in affect or changes in heartbeat rates. Despite the widespread use of ILD, the issues of measurement error and reliability are often inadequately addressed, particularly in the presence of serial correlation and complex multilevel structures. This paper introduces a three-level linear mixed-effects model that explicitly accounts for the nested data structure and the serial correlation (i.e., moments within days, days within persons). The model incorporates two distinct autoregressive processes to capture serial correlations at both the moment and day levels, while simultaneously partitioning measurement error from within-person variability. Building on this model, we delineate two approaches to defining reliability: a variance-partitioning approach and a correlation approach. This framework yields a multifaceted set of reliability coefficients, enabling researchers to distinguish between-person from within-person reliability, and to assess measurement agreement across different time scales (e.g., moment-to-moment and day-to-day). We further discuss the relationship between the proposed reliability measures and those derived from simpler models. We apply our model to study the reliability of a single negative affect item in an ILD study in which students were measured multiple times a day for several days before and after receiving their exam results.