Aaron Marker

Accurate psychological evaluation is crucial for early detection, intervention and evaluation of mental health disorders. The validity and reliability of psychological evaluation methods have been systematically researched for many decades. However, assessment methods used by mental health professionals in clinical settings have not consistently been empirically documented. This study examines the extent to which standardized assessment methods are used in clinical settings, and clinicians’ attitudes toward these practices. Clinicians (N = 585) from the U.S. (25%), the U.K. (28%), the Netherlands (18%), Sweden (22%), and other countries (7%) completed an online survey reporting their depression assessment methods. Clinicians reported using many different data collection methods (e.g., unstructured clinical interviews [83%], rating scales [83%]) in various combinations. A majority (80%) primarily employed clinical data combination methods, as opposed to statistical data combination methods (11%). Although they were confident in their assessments, the time to carry out assessments varied widely (M = 87 [SD = 108] minutes). Seventy-seven percent agreed on the importance of standardized, valid, and reliable assessments, and 43% reported interest in receiving AI decision-support. We did not find many significant cross-cultural differences. A majority of clinicians have not fully adopted standardized assessment methods. Predominant reliance on clinical judgment suggests a disconnect between best practices supported by research and everyday clinical practice.

Introduction: Clinical trial endpoints are a ‘finite sequence of instructions to perform a task’ (measure treatment effectiveness), making them algorithms. Consequently, they may exhibit algorithmic bias: internal and external performance can vary across demographic groups, impacting fairness, validity and clinical decision‐making. Methods: We developed the open‐source Detecting Algorithmic Bias (DAB) Pipeline in Python to identify endpoint ‘performance variance’—a specific algorithmic bias—as the proportion of minority participants changes. This pipeline assesses internal performance (on demographically matched test data) and external performance (on demographically diverse validation data) using metrics including F1 scores and area under the receiver operating characteristic curve (AUROC). We applied it to representative opioid use disorder (OUD) trial endpoints. Results: F1 scores remained stable across minority representation levels, suggesting consistency in precision‐recall balance (F1) despite demographic shifts. Conversely, AUROC measures were more sensitive, revealing significant performance variance. Training on demographically homogeneous populations boosted internal performance (accuracy within similar cohorts) but critically compromised external generalisability (accuracy within diverse cohorts). This pattern reveals an ‘endpoint bias trade‐off’: optimising performance for homogeneous populations vs. having generalisable performance for the real world. Discussion and Conclusions: Consistently performing endpoints for one demographic profile may lose generalisability during population shifts, potentially introducing endpoint bias. Increasing minority representation in the training data consistently improved generalisability. The endpoint bias trade‐off reinforces the importance of diverse recruitment in OUD trials. The DAB Pipeline helps researchers systematically pinpoint when an endpoint may suffer ‘performance variance’ (i.e., bias). As an open‐source tool, it promotes transparent endpoint evaluation and supports selecting demographically invariant OUD endpoints.