The Society for Implementation Research Collaboration Instrument Review Project: A methodology to promote rigorous evaluation
Open Access
- 8 January 2015
- journal article
- research article
- Published by Springer Nature in Implementation Science
- Vol. 10 (1) , 1-18
Abstract
Identification of psychometrically strong instruments for the field of implementation science is a high priority underscored in a recent National Institutes of Health working meeting (October 2013). Existing instrument reviews are limited in scope, methods, and findings. The Society for Implementation Research Collaboration Instrument Review Project’s objectives address these limitations by identifying and applying a unique methodology to conduct a systematic and comprehensive review of quantitative instruments assessing constructs delineated in two of the field’s most widely used frameworks, adopt a systematic search process (using standard search strings), and engage an international team of experts to assess the full range of psychometric criteria (reliability, construct and criterion validity). Although this work focuses on implementation of psychosocial interventions in mental health and health-care settings, the methodology and results will likely be useful across a broad spectrum of settings. This effort has culminated in a centralized online open-access repository of instruments depicting graphical head-to-head comparisons of their psychometric properties. This article describes the methodology and preliminary outcomes. The seven stages of the review, synthesis, and evaluation methodology include (1) setting the scope for the review, (2) identifying frameworks to organize and complete the review, (3) generating a search protocol for the literature review of constructs, (4) literature review of specific instruments, (5) development of an evidence-based assessment rating criteria, (6) data extraction and rating instrument quality by a task force of implementation experts to inform knowledge synthesis, and (7) the creation of a website repository. To date, this multi-faceted and collaborative search and synthesis methodology has identified over 420 instruments related to 34 constructs (total 48 including subconstructs) that are relevant to implementation science. Despite numerous constructs having greater than 20 available instruments, which implies saturation, preliminary results suggest that few instruments stem from gold standard development procedures. We anticipate identifying few high-quality, psychometrically sound instruments once our evidence-based assessment rating criteria have been applied. The results of this methodology may enhance the rigor of implementation science evaluations by systematically facilitating access to psychometrically validated instruments and identifying where further instrument development is needed.Keywords
This publication has 28 references indexed in Scilit:
- Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measuresImplementation Science, 2013
- Advancing the application, quality and harmonization of implementation science measuresImplementation Science, 2012
- A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental HealthMedical Care Research and Review, 2011
- A systematic exploration of differences in contextual factors related to implementing the MOVE! weight management program in VA: A mixed methods studyBMC Health Services Research, 2011
- A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel?Implementation Science, 2010
- Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation scienceImplementation Science, 2009
- Specifying and reporting complex behaviour change interventions: the need for a scientific methodImplementation Science, 2009
- Mapping new theoretical and methodological terrain for knowledge translation: contributions from critical realism and the artsImplementation Science, 2009
- Implementation Research in Mental Health Services: an Emerging Science with Conceptual, Methodological, and Training challengesAdministration and Policy in Mental Health and Mental Health Services Research, 2008
- A national survey of practicing psychologists' attitudes toward psychotherapy treatment manuals.Journal of Consulting and Clinical Psychology, 2000