Medical education research quality (MERSQ) checklist development: Are searches of BEME and non-BEME reviews standard?: A mixed method study
Authors:
- Maryam AlizadehMAMaryam Alizadeh
- Rahem RahmatiRRRahem Rahmati
- Fatemeh ZarimeidaniFZFatemeh Zarimeidani
- Fatemeh HasaniFHFatemeh Hasani
- Arshin GhaediAGArshin Ghaedi
- Aida BazrgarABAida Bazrgar
- Reza Hosseini DoalameRDReza Hosseini Doalame
- Hojat VahediHVHojat Vahedi
- Hamidreza HekmatHHHamidreza Hekmat
- Negar OmidiNONegar Omidi
Open Access
- 2 May 2025
- journal article
- research article
- Published by Wolters Kluwer Health in Medicine
- Vol. 104 (18) , e42316
Abstract
Even though there has been a lot of research in medical education, the quality of it has not increased similarly. This study aimed to provide a valid and reliable user-friendly tool for evaluating search strategies in medical education systematic reviews. This mixed study was conducted in 2019 to 2021, including 3 phases: systematic search, developing a medical education research quality (MERSQ) checklist, and evaluation of the search quality of best evidence in medical education collaboration (BEME) and non-BEME reviews. Three hundred nineteen items were retrieved from the systematic search of PubMed, Embase, Scopus, Psychinfo, ERIC, and Google Scholar. Following ensuring acceptable criteria, 30 items were included in comprehensiveness or reproducibility guarantees. The results showed that the instrument had an the intra-class correlation coefficient of 0.922 (P = .002), the reproducibility guarantee had 0.903 (P = .003), and the comprehensiveness guarantee had 0.926 (P = .006). We also calculated inter-rater reliability and internal consistency using Cronbach alpha of 0.827 (P < .001) and an instrument the intra-class correlation coefficient of 0.978. Using MERSQ, the overall search quality (41.75 vs 31.25, P = .009), reproducibility (22 vs 14.50, P = .004), and comprehensive score (18.75 vs 15.75, P = .880) of BEME studies were higher than non-BEME ones. Moreover, we found only 30% of studies completed searching documents. The search strategy query concerning the selection of synonym terms received the lowest score among studies. This study led to the development of a valid and reliable checklist for evaluating the search quality of medical education systematic reviews. Utilizing the MERSQ checklist, we found that BEME studies had higher quality than non-BEME ones, making the results from BEME studies more reliable.Keywords
This publication has 22 references indexed in Scilit:
- Checklists to reduce diagnostic error: a systematic review of the literature using a human factors frameworkBMJ Open, 2022
- Learning Analytics in Medical Education Assessment: The Past, the Present, and the FutureAEM Education and Training, 2018
- How to Write a Systematic Review of the LiteratureHERD: Health Environments Research & Design Journal, 2017
- Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000-2010): are the searches clearly explained, systematic and reproducible?Research Synthesis Methods, 2013
- Conducting systematic reviews in medical education: a stepwise approachMedical Education, 2012
- Reproducibility of Literature Search Reporting in Medical Education ReviewsAcademic Medicine, 2011
- Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviewsBMC Medical Research Methodology, 2007
- Errors in search strategies were identified by type and frequencyJournal of Clinical Epidemiology, 2006
- Twelve tips for undertaking a systematic reviewMedical Teacher, 2002
- Meta-analysis of Observational Studies in EpidemiologyA Proposal for ReportingJAMA, 2000