Describe the tests of significance and the principles of hypothesis testing.

The Logic of Hypothesis Testing

• The steps, which should be recited in order:
• 1. State the null hypothesis (H0) — that there is NO difference, NO association or NO effect; and the alternative hypothesis (H1) — that there is
• 2. Fix the level of significance (alpha), conventionally 0.05, IN advance
• 3. Choose the appropriate test, based on the type of data, the number of groups, whether the groups are paired, and whether the distribution is normal
• 4. Calculate the test statistic and the degrees of freedom
• 5. Compare with the tabulated value, or obtain the p value
• 6. Draw the inference, and interpret it in clinical terms
• The null hypothesis IS always the hypothesis OF NO difference, and the test can only reject it or fail to reject it — it can never prove it true
• The p value is the probability of obtaining a result AS extreme as, or more extreme than, the one observed, IF the null hypothesis were true
• The misinterpretations of the p value are so universal that stating them earns marks:
• It is not the probability that the null hypothesis is true
• It is not the probability that the result occurred by chance
• A smaller p value does not mean a larger or more important effect — it reflects the strength of the evidence against the null, which depends heavily on sample size
• P > 0.05 does not prove there is no effect; it means the study did not demonstrate one
• The 0.05 threshold is a convention, not a law of nature, and p = 0.049 and p = 0.051 are practically identical despite falling on opposite sides of it
• Statistical significance IS not clinical significance, and this is the single most important interpretive point
• With a large enough sample, a trivial difference becomes statistically significant — a drug lowering systolic pressure by 1 mmHg in 50,000 patients will yield p < 0.001 and help nobody
• Conversely a small study may miss a clinically important difference
• So the questions to ask of any result are, in order — how big is the effect; how precise is the estimate; and does it matter clinically?
• One-tailed versus two-tailed tests — a two-tailed test asks whether there is a difference in either direction and is almost always the correct choice; a one-tailed test is justified only where a difference in one direction is impossible or irrelevant, and choosing it after seeing the data is a form of cheating

Choosing and Applying the Test

• The choice is determined by four questions, and asking them in order reaches the right test reliably: what type of data; how many groups; are the observations paired or independent; and are the data normally distributed?
• Paired versus unpaired is the distinction most often got wrong — data are paired when each observation in one group has a natural partner in the other: the same person before and after, twins, matched cases and controls, or two eyes of the same patient
• Using an unpaired test on paired data wastes the pairing and loses power, because the whole advantage of pairing is that each subject serves as their own control, removing between-person variability
• The assumptions of parametric tests — the data are quantitative; drawn from a normally distributed population; with approximately equal variances in the groups compared; and observations independent of one another
• When these fail, use a non-parametric test, or transform the data
• The problem of multiple comparisons is a major and under-appreciated source of false findings
• If 20 independent comparisons are made at the 5% level, one is expected to be "significant" by chance alone — so testing many subgroups and reporting the one that reached significance is seriously misleading
• The remedies — specify the primary outcome and the planned analyses IN advance; use ANOVA rather than repeated t tests when comparing several groups; apply a correction such as bonferroni; and report all the comparisons made, not merely the successful ones
• Subgroup analyses should be treated as hypothesis-generating, not conclusive, unless they were pre-specified
• Degrees of freedom — broadly the number of values free to vary once the constraints are fixed; (n - 1) for a single sample t test, and (rows - 1) × (columns - 1) for a chi-square contingency table
• The final point to make in any answer on significance testing — a test of significance can tell you whether an effect is likely to be real; it can never tell you whether it matters, whether the study was well designed, or whether the finding is causal. Those judgements remain the reader’s
• Statistical significance is not clinical significance, and a large enough trial makes a trivial effect highly significant
• Ask how big, how precise and does it matter, in that order, of any reported result
• Twenty comparisons at five per cent give one false positive by chance alone, which is why multiple testing must be declared
• Pre-specify the primary outcome, or subgroup findings become hypothesis-generating rather than conclusive
• Paired data need a paired test, or the advantage of each subject serving as their own control is thrown away
• Two eyes of one patient are paired, as are twins and matched case-control pairs, which is often overlooked
• The p value is not the probability the null is true, which is the commonest misinterpretation in medicine
• A smaller p does not mean a bigger effect, only stronger evidence against the null, which depends on sample size
• The 0.05 threshold is a convention, and 0.049 and 0.051 are practically identical results
• Use ANOVA rather than repeated t tests when comparing more than two groups, to avoid inflating the error rate
• Fisher exact test replaces chi-square when expected numbers in any cell are small
• McNemar is the paired version of chi-square, and is used for before-and-after categorical data
• Choose a two-tailed test unless one direction is impossible, and never choose one-tailed after seeing the data
• Ask four questions to choose a test — data type, number of groups, paired or not, and normal or not
• Non-parametric tests suit ordinal and skewed data, and make no assumption about the underlying distribution
• Report all comparisons made, not merely those that reached significance, which is a matter of honesty rather than statistics
• Degrees of freedom are values free to vary once the constraints are fixed, and differ by test
• A test cannot tell you whether a finding is causal, nor whether the study was well designed; those judgements remain the reader