Daniele De Martini

Statistical hypothesis testing (SHT) is widely employed across numerous scientific disciplines, and a clear understanding of its underlying logic is essential for the broader scientific community. Here, drawing upon both epistemological and statistical perspectives, we aim to clarify—primarily for educational purposes—the logical relationship between proof by contradiction (PBC) and SHT. We begin by outlining the logics of SHT and PBC, followed by a discussion of their key similarities and differences. We then explore the pedagogical value of these analogies and distinctions through illustrative examples. Our objective is to help prevent common misinterpretations in the application of SHT, particularly in the interpretation of its outcomes. By elucidating the conceptual parallels between SHT and PBC, we aim to make the logical structure of SHT more transparent. Finally, we highlight and discuss several critical aspects relevant to teaching, with the intention of enhancing the pedagogical effectiveness of instruction in this area.

The large number of failures in phase III clinical trials, which occur at a rate of approximately 45%, is studied herein relative to possible countermeasures. First, the phenomenon of failures is numerically described. Second, the main reasons for failures are reported, together with some generic improvements suggested in the related literature. This study shows how statistics explain, but do not justify, the high failure rate observed. The rate of failures due to a lack of efficacy that are not expected, is considered to be at least 10%. Expanding phase II is the simplest and most intuitive way to reduce phase III failures since it can reduce phase III false negative findings and launches of phase III trials when the treatment is positive but suboptimal. Moreover, phase II enlargement is discussed using an economic profile. As resources for research are often limited, enlarging phase II should be evaluated on a case‐by‐case basis. Alternative strategies, such as biomarker‐based enrichments and adaptive designs, may aid in reducing failures. However, these strategies also have very low application rates with little likelihood of rapid growth.

A quantitative evaluation of individual and collective ethics is proposed here, with the aim of providing a tool for sample size determination/estimation that goes further than the standard power setting of 80–90%. Individual ethics deal with issues that concern the patients enrolled in the trial, where collective ones concern the patients not enrolled in the trial, and who might benefit from a positive result. The global ethical utility (GEU) of a phase III trial is introduced here, being the summation of individual and collective ethical utilities, and can be viewed as a function of the sample size. The GEU model is based on the extent of the efficacy of the treatments in study, of the quality of life of the patients being treated, of the effects of potential adverse reactions, it accounts for the duration of the periods of interest and for the size of population groups, and also embeds the experimental power. This work aims at arguing the case for GEU adoption for sample size determination. The sample size that maximizes GEU can be adopted for planning the trial, even when providing a power value out of the classical range [.8,.9]. Alternatively, among the sample sizes based on power values of 80% and 90%, the one providing the highest GEU can be adopted. Intuitively, when a treatment is assumed to work well, to have few adverse effects, and is expected to improve the QoL of the ill population for a considerable amount of time, collective ethics may prevail giving ethically optimal sample sizes larger than usual, and consequent quite high power values (e.g. 99%). Instead, medium, though still clinically meaningful, levels of effect, considerable adverse reactions, and limited life expectation and QoL improvement, might shift the ethical balance on individual ethics and give an ethically optimal sample size providing a power lower than standard values (e.g. 70%). Some examples and an application in the cardiovascular area, including sensitivity analyses of the results based on the so‐called Bayesian “assurance” technique, are also discussed. Several possible extensions of the model related to particular clinical frameworks are also presented.

Several reproducibility probability (RP)-estimators for the binomial, sign, Wilcoxon signed rank and Kendall tests are studied. Their behavior in terms of MSE is investigated, as well as their performances for RP-testing. Two classes of estimators are considered: the semi-parametric one, where RP-estimators are derived from the expression of the exact or approximated power function, and the non-parametric one, whose RP-estimators are obtained on the basis of the nonparametric plug-in principle. In order to evaluate the precision of RP-estimators for each test, the MSE is computed, and the best overall estimator turns out to belong to the semi-parametric class. Then, in order to evaluate the RP-testing performances provided by RP estimators for each test, the disagreement between the RP-testing decision rule, i.e., “accept H₀ if the RP-estimate is lower than, or equal to, 1/2, and reject H₀ otherwise”, and the classical one (based on the critical value or on the p-value) is obtained. It is shown that the RP-based testing decision for some semi-parametric RP estimators exactly replicates the classical one. In many situations, the RP-estimator replicating the classical decision rule also provides the best MSE.

The reproducibility probability (RP) of a statistically significant outcome is the true power of a statistical test and its estimate is a useful indicator of the stability of the test result. RP-testing consists in testing statistical hypotheses using an RP-estimator as test statistic. In the parametric framework, the RP-based test and the classical one are equivalent, while in the nonparametric one to perform RP-testing is possible only approximately. In this work, we evaluate through a wide simulation study the performances of several semi-parametric and nonparametric RP-estimators (RPEs) for the Wilcoxon rank-sum (WRS) test. RPEs have two tasks: to perform RP-testing and to estimate the RP. To compare RPEs performances we adopt risk indexes (e.g. mean square error (MSE)) and an index of agreement between the outcomes of the WRS test and the RP-based test. Results indicate that the rate of disagreement tends to zero as the sample size increases; the overall rate of disagreement provided by semi-parametric RPEs with finite samples (size 20–200 per group) is 0.15%, and that of nonparametric ones is 0.58%. Concerning risk measures, there is not an RPE dominating the others; for high power values, nonparametric RPEs present the lowest MSE; on average, the semi-parametric RPE based on the upper bound of the variance of the test statistic performs best; nevertheless, the relative gains between the best and the worst are quite small (5–10%). To conclude, well-approximated RP-testing for the WRS test can be performed by adopting a semi-parametric RPE. Since nonparametric plug-in based RPEs perform well in presence of high reproducibility, their adoption is suggested for evaluating the stability of test results and, for example, those of clinical trials.

At least two adequate and well‐controlled clinical studies are usually required to support effectiveness of a certain treatment. In some circumstances, however, a single study providing strong results may be sufficient. Some statistical stability criteria for assessing whether a single study provides very persuasive results are known. A new criterion is introduced, and it is based on the conservative estimation of the reproducibility probability in addition to the possibility of performing statistical tests by referring directly to the reproducibility probability estimate. These stability criteria are compared numerically and conceptually. This work aims to help both regulatory agencies and pharmaceutical companies to decide if the results of a single study may be sufficient to establish effectiveness. Copyright © 2012 John Wiley & Sons, Ltd.

A study on the robustness of the adaptation of the sample size for a phase III trial on the basis of existing phase II data is presented—when phase III is lower than phase II effect size. A criterion of clinical relevance for phase II results is applied in order to launch phase III, where data from phase II cannot be included in statistical analysis. The adaptation consists in adopting the conservative approach to sample size estimation, which takes into account the variability of phase II data. Some conservative sample size estimation strategies, Bayesian and frequentist, are compared with the calibrated optimal γ conservative strategy (viz. COS) which is the best performer when phase II and phase III effect sizes are equal. The Overall Power (OP) of these strategies and the mean square error (MSE) of their sample size estimators are computed under different scenarios, in the presence of the structural bias due to lower phase III effect size, for evaluating the robustness of the strategies. When the structural bias is quite small (i.e., the ratio of phase III to phase II effect size is greater than 0.8), and when some operating conditions for applying sample size estimation hold, COS can still provide acceptable results for planning phase III trials, even if in bias absence the OP was higher. Main results concern the introduction of a correction, which affects just sample size estimates and not launch probabilities, for balancing the structural bias. In particular, the correction is based on a postulation of the structural bias; hence, it is more intuitive and easier to use than those based on the modification of Type I or/and Type II errors. A comparison of corrected conservative sample size estimation strategies is performed in the presence of a quite small bias. When the postulated correction is right, COS provides good OP and the lowest MSE. Moreover, the OPs of COS are even higher than those observed without bias, thanks to higher launch probability and a similar estimation performance. The structural bias can therefore be exploited for improving sample size estimation performances. When the postulated correction is smaller than necessary, COS is still the best performer, and it also works well. A higher than necessary correction should be avoided.