John Ludbrook

SUMMARY: 1. Clinical and experimental pharmacologists and physiologists often wish to compare two methods of measurement, or two measurers.2. Biostatisticians insist that what should be sought is not agreement between methods or measurers, but disagreement or bias.3. If measurements have been made on a continuous scale, the main choice is between the Altman–Bland method of differences and least products regression analysis. It is argued that although the former is relatively simple to execute, it does not distinguish adequately between fixed and proportional bias. Least products regression analysis, although more difficult to execute, does achieve this goal. There is almost universal agreement among biostatisticians that the Pearson product–moment correlation coefficient (r) is valueless as a test for bias.4. If measurements have been made on a categorical scale, unordered or ordered, the most popular method of analysis is to use the kappa statistic. If the categories are unordered, the unweighted kappa statistic (K) is appropriate. If the categories are ordered, as they are in most rating scales in clinical, psychological and epidemiological research, the weighted kappa statistic (K_w) is preferable. But K_w corresponds to the intraclass correlation coefficient, which, like r for continuous variables, is incapable of detecting bias. Simple techniques for detecting bias in the case of ordered categorical variables are described and commended to investigators.

SUMMARY: 1. Learning how to apply statistical analyses to the results of experimental or clinical studies may take a lifetime of trial (and sometimes error), as it has done in the author’s case. There is no evidence that biomedical investigators of the present generation are on a steeper learning curve. Gross misunderstandings of the purpose and functions of statistical analysis are apparent in applications to research grant‐giving bodies and ethics committees, in manuscripts submitted to journals and sometimes in published papers.2. Although estimation of minimal group (sample) size for a given power is an essential step in planning clinical studies, it seems to be used rarely in laboratory experimental work. This is despite exhortations to restrict the number of animals used to a minimum.3. Most investigators use hypothesis testing to analyse their results, but their understanding of the meaning of the resultant P‐values is slight.4. A flaw found almost universally in biomedical manuscripts is to make multiple inferences from the results of a single study. The goal of statistical analysis is to maintain the familywise type I error rate (risk of false‐positive inference) at a predetermined level (usually 5%). But, when multiple inferences are made from the same experiment, the risk of false‐positive error is inflated. There are two solutions to this problem: (i) use a multiple comparison procedure to control the familywise type I error rate; and (ii) test a single, global hypothesis.5. Biomedical investigators have been quick to acquire computer statistics software and to use it to analyse their experiments. However, they have been slow to recognize the limitations of this software. These include: (i) inadequate documentation of routines, so that neither the user nor the reader of published papers can be sure how the tests have been executed; (ii) flawed algorithms for the execution of statistical procedures; and (iii) failure to recognize that the best software for their purposes is that which takes them just beyond their statistical horizons.6. The obvious solution to these difficulties is to recruit a biomedical statistician into every research group, at a relatively trivial cost. However, properly qualified biostatisticians are in desperately short supply in Australia. It follows that research groups, national grant‐giving agencies and academic institutions must make provision for the proper training and subsequent employment of biostatisticians.

SUMMARY: 1. In a recent review article, the problem of making false‐ positive inferences as a result of making multiple comparisons between groups of experimental units or between experimental outcomes was addressed.2. It was concluded that the most universally applicable solution was to use the Ryan–Holm step‐down Bonferroni procedure to control the family‐wise (experiment‐wise) type 1 error rate. This procedure consists of adjusting the P values resulting from hypothesis testing. It allows for correlation among hypotheses and has been validated by Monte Carlo simulation. It is a simple procedure and can be performed by hand.3. However, some investigators prefer to estimate effect sizes and make inferences by way of confidence intervals rather than, or in addition to, testing hypotheses by way of P values and it is the policy of some editors of biomedical journals to insist on this. It is not generally recognized that confidence intervals, like P values, must be adjusted if multiple inferences are made from confidence intervals in a single experiment.4. In the present review, it is shown how confidence intervals can be adjusted for multiplicity by an extension of the Ryan–Holm step‐down Bonferroni procedure. This can be done for differences between group means in the case of continuous variables and for odds ratios or relative risks in the case of categorical variables set out as 2 × 2 tables.

SUMMARY: 1. A common statistical flaw in articles submitted to or published in biomedical research journals is to test multiple null hypotheses that originate from the results of a single experiment without correcting for the inflated risk of type 1 error (false positive statistical inference) that results from this. Multiple comparison procedures (MCP) are designed to minimize this risk. The present review focuses on pairwise contrasts, the most common sort of multiple comparisons made by biomedical investigators.2. In an earlier review a variety of MCP were described and evaluated. It was concluded that an effective MCP should control the risk of family‐wise type 1 error, so as to ensure that not more than one hypothesis within a single family is falsely rejected. One‐step procedures based on the Bonferroni or Sidak inequalities do this. For continuous data and under normal distribution theory, so does the Tukey‐Kramer procedure for all possible pairwise contrasts of means and the Dunnett procedure for all possible pairwise contrasts of means with a control mean.3. There is now a new class of MCP, based on the Bonferroni or Šidák inequalities but performed in a step‐wise fashion. The members of this class have certain desirable properties. They: (i) control the family‐wise type 1 error rate as effectively as the one‐step procedures; (ii) are more powerful than the one‐step Bonferroni or Sidak procedures, especially when hypotheses are logically related; and (iii) can be applied not only to continuous data but also to ordinal or categorical data.4. Of the new step‐wise MCP, Holm's step‐down procedures are commended for their combination of accuracy, power and versatility. They also have the virtue of simplicity. Given the raw P values that result from conventional tests of significance, the adjustments for multiple comparisons can be made by hand or hand‐held calculator.5. Despite the corrective abilities of the new step‐wise MCP, investigators should try to design their experiments and analyses to test a single, global hypothesis rather than multiple ones.