Validating the knowledge bank approach for personalized prediction of survival in acute myeloid leukemia: a reproducibility study
Open Access
- 16 April 2022
- journal article
- research article
- Published by Springer Nature in Human Genetics
- Vol. 141 (9) , 1467-1480
Abstract
Reproducibility is not only essential for the integrity of scientific research but is also a prerequisite for model validation and refinement for the future application of predictive algorithms. However, reproducible research is becoming increasingly challenging, particularly in high-dimensional genomic data analyses with complex statistical or algorithmic techniques. Given that there are no mandatory requirements in most biomedical and statistical journals to provide the original data, analytical source code, or other relevant materials for publication, accessibility to these supplements naturally suggests a greater credibility of the published work. In this study, we performed a reproducibility assessment of the notable paper by Gerstung et al. (Nat Genet 49:332–340, 2017) by rerunning the analysis using their original code and data, which are publicly accessible. Despite an open science setting, it was challenging to reproduce the entire research project; reasons included: incomplete data and documentation, suboptimal code readability, coding errors, limited portability of intensive computing performed on a specific platform, and an R computing environment that could no longer be re-established. We learn that the availability of code and data does not guarantee transparency and reproducibility of a study; paradoxically, the source code is still liable to error and obsolescence, essentially due to methodological and computational complexity, a lack of reproducibility checking at submission, and updates for software and operating environment. The complex code may also hide problematic methodological aspects of the proposed research. Building on the experience gained, we discuss the best programming and software engineering practices that could have been employed to improve reproducibility, and propose practical criteria for the conduct and reporting of reproducibility studies for future researchers.Keywords
All Related Versions
Funding Information
- Ludwig-Maximilians-Universität München
This publication has 39 references indexed in Scilit:
- How to Make More Published Research TruePLOS Medicine, 2014
- Cox models with dynamic ridge penalties on time‐varying effects of the covariatesStatistics in Medicine, 2013
- Genomic and Epigenomic Landscapes of Adult De Novo Acute Myeloid LeukemiaNew England Journal of Medicine, 2013
- Case studies in reproducibilityBriefings in Bioinformatics, 2011
- Prospective Evaluation of Allogeneic Hematopoietic Stem-Cell Transplantation From Matched Related and Matched Unrelated Donors in Younger Adults With High-Risk Acute Myeloid Leukemia: German-Austrian Trial AMLHD98AJournal of Clinical Oncology, 2010
- The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational StudiesAnnals of Internal Medicine, 2007
- Reproducible Research: A Bioinformatics Case StudyStatistical Applications in Genetics and Molecular Biology, 2005
- Phase III study of all-trans retinoic acid in previously untreated patients 61 years or older with acute myeloid leukemiaLeukemia, 2004
- Penalized Survival Models and FrailtyJournal of Computational and Graphical Statistics, 2003
- Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)Statistical Science, 2001