Adam Zweifach

Statistical tests can be used to help determine whether experimental manipulations produce effects. In tests of means, when more than two groups are compared the total number of Type 1 errors (false positive results) increases unless a correction is used. Tukey’s test is thought to offer good control of the false positive rate and high statistical power when all pairwise comparisons are made. However, the number of replicates in laboratory experiments is often quite low, and small sample sizes can undermine assumptions underlying statistical methods. I used simulations to investigate how well Tukey’s test controls the total number of false positives when there are 3-6 experimental groups and 2-6 experimental replicates, conditions that span the range of typical values, and found that it generates too many. I investigated 11 other approaches to controlling false positives and found that none is as effective as the simple Bonferroni correction or offers much more power. I conclude that researchers should not make all pairwise comparisons using Tukey’s test but instead use Bonferroni’s correction on a limited number of pre-selected comparisons.

From a statistical standpoint, individual cells are typically not independent experimental replicates. To test for differences in mean, cells from each experimental sample can be averaged and each sample’s average treated as an n of 1. Here, I outline how to determine how many cells to average per sample.

Power of statistical tests for differences in means is the probability of obtaining a statistically significant p value when means differ. When samples in experimental replicates come from a single cell culture, they are matched or paired because they share between-trials biological variability. This can cause positive correlation between values from conditions in a replicate. Correlation can also be caused in otherwise independent samples by shared technical variability. However, correlation is reduced by noise that affects samples individually. I investigated how to maximize power in experiments with two conditions over a range of correlations. Normalizing data to control increases the rate of false positives, if Student’s t test is used. Paired t tests, theoretically the correct test for matched samples, have higher power than Student’s t test when correlation is high, but lower power when correlation is low. Testing correlation to select a test for differences in mean can affect the subsequent rate of false positives. Ultimately, components of experimental variability must be considered to choose the most powerful two sample test for differences in mean. This contrasts with experiments with more than two conditions, where random-block ANOVA, a matched samples test, can be used as a default.