Methodological quality in randomised clinical trials of mental health apps: systematic review and longitudinal analysis
Open Access
- 12 April 2025
- journal article
- review article
- Published by BMJ in BMJ Mental Health
- Vol. 28 (1) , e301595
Abstract
Question: This study investigated the methodological rigour of randomised controlled trials (RCTs) of mental health apps for depression and anxiety, and whether quality has improved over time. Study selection and analysis: RCTs were drawn from the most recent meta-analysis of mental health apps for depression and anxiety symptoms. 20 indicators of study quality were coded, encompassing risk of bias, participant diversity, study design features and app accessibility measures. Regression models tested associations between year of publication and each quality indicator. Findings: 176 RCTs conducted between 2011 and 2023 were included. Methodological concerns were common for several quality indicators (eg, <20% were replication trials, <35% of trials reported adverse events). Regression models revealed only three significant changes over time: an increase in preregistration (OR=1.27; 95% CI 1.10, 1.46) and reporting of adverse events (OR=1.32; 95% CI 1.11, 1.56), and a decrease in apps reported to be compatible with iOS and/or Android (OR=0.78; 95% CI 0.64, 0.96). Results were unchanged when excluding outliers. Results were similar when excluding three high-quality studies published between 2011 and 2013, with additional evidence for an increase in modern missing data methods (OR=1.22; 95% CI 1.04, 1.42) and studies reporting intention-to-treat analysis (OR=1.20; 95% CI 1.03, 1.39). Conclusions: Findings provide minimal evidence of improvements in the quality of clinical trials of mental health apps, highlighting the need for higher methodological standards in future research to ensure the reliability and generalisability of evidence for these digital tools.Keywords
Funding Information
- NCCIH (K23AT010879)
- NCCIH (R24AT012845)
- National Institute of Mental Health (MH018931)
- National Center for Complementary & Integrative Health (R24AT012845)
- National Health and Medical Research Council (APP1196948)
- National Center for Complementary & Integrative Health (K23AT011173)
This publication has 44 references indexed in Scilit:
- Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior SupportAnnals of Behavioral Medicine, 2017
- The efficacy of smartphone‐based mental health interventions for depressive symptoms: a meta‐analysis of randomized controlled trialsWorld Psychiatry, 2017
- Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trialsJournal of Affective Disorders, 2017
- Does gamification increase engagement with online programs? A systematic reviewPLOS ONE, 2017
- Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesisTranslational Behavioral Medicine, 2016
- Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studiesBMC Medical Informatics and Decision Making, 2016
- Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?American Psychologist, 2015
- A Systematic Review of the Impact of Adherence on the Effectiveness of e-TherapiesJournal of Medical Internet Research, 2011
- The effects of psychotherapy for adult depression are overestimated: a meta-analysis of study quality and effect sizePsychological Medicine, 2009
- The Law of AttritionJournal of Medical Internet Research, 2005