Development of a tool to assess the risk of bias in statistical simulation studies: study protocol
Open Access
- 30 June 2026
- journal article
- Published by F1000 Research Ltd in F1000Research
- Vol. 15, 1038
Abstract
Background Statistical simulation studies are the principal methodology for examining the performance of statistical methods. Findings from statistical simulation studies guide researchers in their statistical decision making, and inform evidence syntheses of statistical simulation studies, so should provide a fair representation of how the methods are expected to perform. Objectives To develop a tool to assess whether statistical simulation studies provide a fair representation of how statistical methods are expected to perform. Methods We will undertake a multi-step process to develop a domain-based tool with signalling questions. The project team will consist of a core working group and an international advisory group. We will hold virtual meetings with the advisory group to agree on the scope and content of the tool. We will conduct systematic reviews and cited reference searches (forward and backward citations) to develop an evidence base to inform domains and signalling questions. An initial set of domains will be agreed on with the advisory group. We will then undertake a survey of methodologists and statisticians with expertise in statistical simulation design and intended users of the tool, to seek their views on which domains are most important. We will propose signalling questions for each domain and revise the domains with feedback from the advisory group until domains are agreed. We will pilot the tool with intended users such as consulting statisticians and systematic reviewers of findings from simulation studies. The developed tool and guidance documentation will be published in an open-access journal and disseminated via conferences and workshops. This protocol has been registered on the Open Science Framework (OSF) on June 2 2026, (Registration DOI: https://doi.org/10.17605/OSF.IO/DW4SM). 1Keywords
Funding Information
- National Health and Medical Research Council (GNT2009612)
- National Health and Medical Research Council (GNT2033917)
This publication has 43 references indexed in Scilit:
- Editorial for the special collection “Towards neutral comparison studies in methodological research”Biometrical Journal, 2024
- A novel causal mediation analysis approach for zero‐inflated mediatorsStatistics in Medicine, 2023
- Explaining the optimistic performance evaluation of newly proposed methods: A cross‐design validation experimentBiometrical Journal, 2023
- Phases of methodological research in biostatistics—Building the evidence base for new methodsBiometrical Journal, 2023
- The Bayesian simulation study (BASIS) framework for simulation studies in statistical and methodological researchBiometrical Journal, 2023
- Over‐optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their resultsWIREs Data Mining and Knowledge Discovery, 2021
- Using simulation studies to evaluate statistical methodsStatistics in Medicine, 2019
- On the necessity and design of studies comparing statistical methodsBiometrical Journal, 2017
- Proposed best practice for projects that involve modelling and simulationPharmaceutical Statistics, 2016
- The design of simulation studies in medical statisticsStatistics in Medicine, 2006