Anne-Laure Boulesteix

A fundamental challenge in drug development is the frequent failure of early laboratory research to translate into clinical benefit. One promising solution is to confirm findings from exploratory single-laboratory studies across multiple laboratories before clinical testing. We investigated this approach following the conduct of preclinical multi-laboratory studies across different fields of medicine. For this, we evaluated effect sizes, experimental rigor, and a set of criteria to identify determinants of confirmation success. When tested under increased rigor, only a fraction of multi-laboratory studies confirmed the initial results. The underlying effect size reduction was associated with outcome-relevant experimental differences between exploratory and confirmatory stages. In summary, multi-laboratory studies proved highly informative and served as an effective filter for promising treatments. One Sentence Summary: Confirmatory testing of exploratory findings in rigorous multi-laboratory settings reveals which hold up and what limits their translation.

Simulation studies are indispensable for evaluating statistical methods and ubiquitous in statistical research. The most common simulation approach is parametric simulation, where the data‐generating mechanism (DGM) corresponds to a parametric model from which observations are drawn. While many simulation studies aim to give practical guidance on method suitability, parametric simulations in particular are often criticized for being unrealistic. To overcome this drawback, it is sensible to employ real data for constructing the parametric DGMs. However, although real‐data‐based parametric DGMs are widely used, the specific ways in which DGM components are inferred vary, and their implications may not be fully understood. Additionally, researchers often rely on a limited selection of real datasets, with the rationale for their selection being unclear. This paper reviews and formally discusses how components of parametric DGMs can be inferred from real data and how dataset selection can be performed more systematically. By doing so, we aim to support researchers in conducting simulation studies with a lower risk of overgeneralization and misinterpretation. We illustrate the construction of parametric DGMs based on a systematically selected set of real datasets using two examples: one on ordinal outcomes in randomized controlled trials and one on differential gene expression analysis.

Estimating the generalization error (GE) of machine learning models is fundamental, with resampling methods being the most common approach. However, in nonstandard settings, particularly those where observations are not independently and identically distributed, resampling using simple random data divisions may lead to biased GE estimates. This paper strives to present well-grounded guidelines for GE estimation in various such nonstandard settings: clustered data, spatial data, unequal sampling probabilities, concept drift and hierarchically structured outcomes. Our overview combines well-established methodologies with other existing methods that, to our knowledge, have not been frequently considered in these particular settings. A unifying principle among these techniques is that the test data used in each iteration of the resampling procedure should reflect the new observations to which the model will be applied, while the training data should be representative of the entire data set used to obtain the final model. Beyond providing an overview, we address literature gaps by conducting simulation studies and a real data study. These studies assess the necessity of using GE-estimation methods tailored to the respective setting. Our findings corroborate the concern that standard resampling methods often yield biased GE estimates in nonstandard settings, underscoring the importance of tailored GE estimation.

Background: Methods research aims to develop, evaluate, refine, or provide guidance on research methods, support their implementation or de-implementation, or synthesize existing methods research. Many types of methods research appear in health research journals, but their labels, definitions, and boundaries remain unclear. This complicates planning, reporting, indexing, and teaching, and limits the visibility of the field. We aimed to develop, refine and pilot-test a typology for recognising, describing, and classifying methods research.Methods: An international panel of 11 methodologists developed the typology through written feedback, structured group discussions, and iterative revision. In two testing rounds, panel members independently applied the typology to 52 methods research articles from 26 journals, generating 494 article assessments. We used the results to refine definitions and labels, add boundary notes, and document remaining disagreements.Results: TYPE-ME defines methods research and organizes it along two cross-cutting dimensions: approach (“how?”, 15 types) and objective (“why?”, 14 types). Testing clarified difficult boundaries and led to a condensed version that distinguishes 10 types for quick orientation: theory-based methods research; simulation studies; methods research based on real data; methodological randomized studies; surveys, interviews, or focus groups; consensus studies; studies of applied methods; syntheses of methods research; guidance for applying, reporting, or assessing methods; and methodological implementation research. Detailed type descriptions, boundary notes, and examples appear in an accompanying document and at www.methods-research.org.Conclusions: TYPE-ME offers a common reference for recognising, describing, and classifying methods research, and can support its planning, reporting, indexing, and teaching. As a living typology, it will evolve through application, user feedback, and further testing.

Confirmatory multi‐lab preclinical trials are a powerful experimental strategy to enable decisions to transition from preclinical to clinical settings. With their complexity, such study designs pose several challenges in statistical planning, analysis, and reporting of experiments. To address these, we convened an expert group of biostatisticians and biomedical scientists currently involved in such trials to summarize in a tutorial the most common challenges and offer general guidance. Furthermore, we incorporated statistical advice from existing clinical trials’ guidelines and adapted it into recommendations for preclinical trials. We describe strategies on key topics such as calculating sample sizes, handling differences between centers, and selecting relevant covariates. Additionally, we give guidance on statistical methods to account for lab effects and proper reporting of analyses. We embed this in a general discussion on remaining open questions to advance the analysis of preclinical confirmatory studies. The provided general, non‐case‐specific guidance serves as a conversation starter between biomedical scientists and statisticians to develop robust statistical analysis strategies for confirmatory multi‐lab preclinical trials.

Most original articles published in the medical literature report the results of multiple statistical tests. In a few simple cases, there is agreement on whether to adjust for the number of performed tests. For many cases encountered in practice, however, this is less clear, and the recommendations in the literature are contradictory, along different dimensions, or otherwise confusing. This lack of clear guidance may impair the conduct and interpretation of analyses, and encourage questionable research practices, ultimately jeopardizing the credibility of medical research. In this article, we refine, illustrate, and discuss a unifying guiding principle to assist both statisticians and applied researchers in deciding whether to adjust for multiple testing and, if so, over which set of tests. The principle is that multiple testing should be adjusted for if and only if authors, when reporting and interpreting their findings, put more emphasis on results of one or several of the tests because of their small p‐value(s) . We relate this principle to previously proposed rules and show how it can guide and clarify the choice of adjustment strategies in three complex multiple testing settings.

Summary: Background: Anthropometric measures do not adequately capture heterogeneity in body fat distribution and corresponding cardiometabolic risk, whereas magnetic resonance imaging (MRI) enables precise differentiation and quantification of adipose tissue compartments and ectopic fat. We aimed to validate previously derived MRI-based body composition subphenotypes and their cardiometabolic risk profiles in two independent European cohorts. Methods: Using deep learning–based image analysis, we quantified bone marrow, visceral, subcutaneous, cardiac, renal sinus, hepatic, skeletal muscle, and pancreatic fat in the imaging substudies of two population-based cohorts: the German National Cohort (NAKO, N=29,314, age range 19-74 years) and the UK Biobank (N=36,109, age range 40-69 years). Body composition subphenotypes, previously identified by k-means clustering, were evaluated using a rigorous statistical cluster validation framework with method-based and results-based approaches. In NAKO, cross-sectional associations between subphenotypes and estimated cardiovascular disease risk scores were examined using linear regression. In UK Biobank, longitudinal associations between subphenotypes and incident cardiometabolic outcomes, ascertained through hospital record linkage, were analysed using Cox regression. Findings: All five body composition subphenotypes were robustly validated across both cohorts, and showed distinct fat distribution patterns and cardiometabolic risk profiles: I “lean”, II “average adiposity”, III “bone and muscle adiposity”, IV “hepato-abdominal adiposity”, and V “general and pancreatic adiposity”. Subphenotypes I–III showed progressive adipose tissue remodelling patterns likely reflecting ageing trajectories. The “hepato-abdominal adiposity” subphenotype showed highest risk of incident diabetes, whereas the “general and pancreatic adiposity” subphenotype showed highest overall cardiovascular disease burden and metabolic impairment. Interpretation: MRI-derived body composition subphenotypes represent distinct fat distribution patterns that reflect ageing- and disease-related processes, which supports the potential of body composition phenotyping for improved cardiometabolic risk stratification and targeted prevention.

Using real data examples from methodological research, Maximilian Mandl, Mathilde Dicaire-Cartier, Tobias Woehrle, Frank Weber and Anne-Laure Boulesteix highlight the perils of presenting a method's performance in the most flattering light using real data examples

Computational reproducibility, the possibility for independent researchers to exactly reproduce published empirical results, is fundamental to science. Despite its importance, the proportion of research articles aiming for reproducibility remains low and uneven across disciplines. Barriers include a perceived lack of incentives for researchers and journals, practical challenges in preparing reproducible materials, and the absence of harmonized standards of reproducibility processes and requirements by journals. Existing guidance is often highly technical, reaching mainly those already engaged with reproducible research. In this paper, we first synthesize evidence on the benefits of reproducibility for both authors and journals. Drawing on our extensive experience in reproducibility checking at various journals, we then put forward concise, pragmatic guidelines and the REPRO Checklist for creating reproducible analyses across disciplines. We further review current reproducibility policies of selected journals, illustrating the substantial heterogeneity in requirements and procedures. Motivated by the latter, we propose conceptual foundations for a harmonized multi-tier system of reproducibility standards that could support transparent, consistent assessment across journals and research communities. Our goal as journal (reproducibility) editors and contributors to the Mathematical Research Data Initiative is to encourage broader adoption of reproducibility practices, in particular by lowering practical barriers for authors and journals.

Introduction: Health researchers benefit from methods guidance (MG) for planning, conducting, analyzing and interpreting research studies. How to develop MG that is trustworthy and researcher-friendly is unclear; existing strategies for developing MG are fragmented and lack a coherent framework. The STrategies for developing REseArch Methods guidance (STREAM) project aims to provide such a framework for developers of MG. As a first step, we reviewed the literature to identify proposed and applied strategies for developing MG and explored possible knowledge gaps.Methods: We conducted two literature reviews. First, a scoping review of the methodological literature to identify proposed strategies for developing MG. Second, a review of a purposeful sample of recently published MG articles to examine strategies applied in practice. We applied a qualitative content analysis to both sets of publications using one shared codebook. In addition, to identify possible knowledge gaps, we contrasted the findings with the established development processes for clinical practice guidelines and reporting guidelines.Results: The first review included 17 publications and the second review 102 MG articles. We identified possible development steps including assessing the need for MG, establishing a working group, considering the findings from methodological studies, gathering input from interest-holders, testing the new MG, and disseminating and implementing MG. In addition, we identified various approaches to perform the steps including systematic reviews of previous guidance, reviews of methodological practice, simulation studies, surveys or interviews of interest-holders, consensus methods, and public review. Mapping these findings with established development processes identified areas that have so far received limited attention within MG development, including the concept of methodological evidence, judging the quality of methodological evidence, reporting of MG, and approaches to implementation.Conclusion: This study provides an overview of proposed and applied strategies for developing MG, highlights needs for conceptual development, and provides a structured starting point for guidance for developing MG.