What is meta-analysis?

Meta-analysis is the use of statistical methods to combine results of individual studies. This allows us to make the best use of all the information we have gathered in our systematic review by increasing the power of the analysis. By statistically combining the results of similar studies we can improve the precision of our estimates of treatment effect, and assess whether treatment effects are similar in similar situations. The decision about whether or not the results of individual studies are similar enough to be combined in a meta-analysis is essential to the validity of the result, and will be covered in the next module on heterogeneity. In this module we will look at the process of combining studies and outline the various methods available.

There are many approaches to meta-analysis. We have discussed already that meta-analysis is not simply a matter of adding up numbers of participants across studies (although unfortunately some non-Cochrane reviews do this). This is the 'pooling participants' or 'treat-as-one-trial' method and we will discuss it in a little more detail now.

Pooling participants (not a valid approach to meta-analysis).

This method effectively considers the participants in all the studies as if they were part of one big study. Suppose the studies are randomised controlled trials: we could look at everyone who received the experimental intervention by adding up the experimental group events and sample sizes and compare them with everyone who received the control intervention. This is a tempting way to 'pool results', but let's demonstrate how it can produce the wrong answer.

A Cochrane review of trials of daycare for pre-school children included the following two trials. For this example we will focus on the outcome of whether a child was retained in the same class after a period in either a daycare treatment group or a non-daycare control group. In the first trial (Gray 1970), the risk difference is -0.16, so daycare looks promising:

In the second trial (Schweinhart 1993) the absolute risk of being retained in the same class is considerably lower, but the risk difference, while small, still lies on the side of a benefit of daycare:

What would happen if we pooled all the children as if they were part of a single trial?

It suddenly looks as if daycare may be harmful: the risk difference is now bigger than 0. This is called Simpson's paradox (or bias), and is why we don't pool participants directly across studies. The first rule of meta-analysis is to keep participants within each study grouped together, so as to preserve the effects of randomisation and compare like with like. Therefore, we must take the comparison of risks within each of the two trials and somehow combine these. In practice, this means we need to calculate a single measure of treatment effect from each study before contemplating meta-analysis. For example, for a dichotomous outcome (like being retained in the same class) we calculate a risk ratio, the risk difference or the odds ratio for each study separately, then pool these estimates of effect across the studies.

Simple average of treatment effects (not used in Cochrane reviews)
If we obtain a treatment effect separately from each study, what do we do with them in the meta-analysis? How about taking the average? The average of the risk differences in the two trials above is (-0.004 - 0.16) / 2 = - 0.082. This may seem fair at first, but the second trial randomised more than twice as many children as the first, so the contribution of each randomised child in the first trial is diminished. It is not uncommon for a meta-analysis to contain trials of vastly different sizes. To give each one the same influence cannot be reasonable. So we need a better method than a simple average.