Skip navigation
7 Simple Comparisons
- When we compare groups or factors we should choose a statistic or point estimate that captures or quantifies that comparison. For example, if we have a binary outcome, we may choose a risk ratio or odds ratio, if we have a continuous outcome, then we may compare groups by calculating the mean difference. We call these types of point estimates, measures of effect.
- It is important to check the assumptions required for obtaining a valid confidence interval or hypothesis test. The most important two are that the observations are independent and that the sample size is sufficient (if using a normal approximation). If the assumptions are violated then an alternative approach should be sought otherwise the inference may lack internal validity and lead to incorrect conclusions.
- We use the t-distribution for comparing means when we have small sample sizes. The t-distribution accounts for the extra uncertainty in estimating the population standard deviation which is needed to estimate the standard error of the sampling distribution. Because the t-distribution is used for small sample sizes, we also have to make an extra assumption that the parent population from where the sample came from is normally distributed.
- The null and alternative hypotheses must be stated (and thought about) in terms of population parameters because the hypothesis concerns what happens in the population, the data are used as evidence against the hypothesis.
- Odds ratios approximate risk ratios when the outcome is fairly rare (<10%), nonetheless, it is incorrect to interpret them as a risk ratio.
- Odds ratios and risk ratios are relative measures of association, they describe the relative change in odds or risk from being exposed versus not exposed. Because they are a relative measure they are sometimes preferred as an indicator of the strength of association for comparing proportions over the risk difference. The risk difference measures the association between an exposure and outcome in terms of the absolute change in risk.
- In population based studies it can be misleading to present the risk ratio without also reporting the risk difference as it can exagerate the risk when the baseline risk is very low. For example consider a background risk of 0.2% which is increased to 0.5% from exposure to a factor; this is a risk ratio of 2.5 or a 250% relative increase in risk but only a 0.3% increase in absolute risk.
- Understanding and exploring data:
- Sometimes you can just look at the data and see the null hypothesis is clearly false. (For example, if a histogram shows that every observation is greater than the null hypothesis value). The methods for simple comparisons are a way of formalizing what you see.
- In order to evaluate whether certain assumptions hold for confidence intervals and hypothesis tests (eg, for t-tests) we sometimes have to look at the shape, centre and spread of the sample distribution.
- Probability:
- A p-value is a conditional probability; given the null hypothesis is true, what is the probability of getting the observed statistic (or something more extreme)?
- Statistical inference & data collection:
- Methods using the normal distribution rely on sampling distributions and the Central Limit Theorem. For assumptions to be met, we also need to follow good data collection and study design practices to obtain a random sample and minimise bias.
- Statistical modelling:
- All of the parametric simple comparisons involve a statistical model, for example, we assume sampling distribution of each follows a model with some hypothesized mean.
- Appraisal of research:
- The simple comparisons and measures of effect covered in this theme are widely used in the published literature. Understanding these methods thus allows you to assess whether the results of a study support the conclusions made by the authors and so is an important aspect of critical appraisal.