A Python module to combine p values arising from discrete tests.
Open Access
- 29 February 2024
- journal article
- Published by The Open Journal in Journal of Open Source Software
- Vol. 9 (94) , 6096
Abstract
Most popular hypothesis tests are designed for datasets with a simple structure. For example, in a simple medical trial, this structure would be a treatment and a control group, and a test would answer whether some health score exhibits a significant difference between their members. However, many datasets are naturally segmented or benefit from being analysed in segments. Continuing our example, if we know a priori that females score higher than males on average, analysing each sex separately should allow us to see more clearly whether the treatment is effective. To draw an overall conclusion in such a case, we need to combine the 𝑝 values for each sub-dataset. Starting with Fisher's method, several ways have been proposed to combine 𝑝 values (Heard & Rubin-Delanchy, 2018). Their usual implementations assume continuous tests, i.e., tests whose 𝑝 values under the null hypothesis follow the uniform distribution on the unit interval. However, rank tests and many others are discrete tests, i.e., they can only yield 𝑝 values from a finite selection (for a given sample size). Assuming continuous tests when combining 𝑝 values from discrete tests can lead to considerably misleading outcomes (Kincaid, 1962; Mielke et al., 2004), in particular when the sub-datasets are small. We here present the Python module combine_pvalues_discrete, which provides a toolbox for combining 𝑝 values from discrete tests. Results from individual tests are stored with the respective null distribution of 𝑝 values and the combined 𝑝 value is accurately estimated using a Monte Carlo simulation based on these null distributions.Keywords
This publication has 6 references indexed in Scilit:
- The poolr Package for Combining Independent and Dependent p Values2022
- Choosing between methods of combining $p$-valuesBiometrika, 2018
- metap: Meta-Analysis of Significance Values2014
- Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly DrawnStatistical Applications in Genetics and Molecular Biology, 2010
- Combining Probability Values from Independent Permutation Tests: A Discrete Analog of Fisher's Classical MethodPsychological Reports, 2004
- The Combination of Tests Based on Discrete DistributionsJournal of the American Statistical Association, 1962