Variety is the spice of life

Systematic reviews usually bring together studies that were performed

• by different people
• in different settings
• in different countries
• on different people
• in different ways
• for different lengths of time
• to look at different outcomes

… and these aren't the only differences.

However, while studies are never the same, they may all have similar results. In fact, the purpose of a Cochrane review is to collate studies that are similar. The decision to combine studies in a meta-analysis in your review is a judgement you will have to make, based on your knowledge about how differences between studies might influence how effective a treatment is observed to be. Sometimes studies are similar enough to consider performing a meta-analysis; sometimes they are not. What we can then do is look at the results of the studies we find to see if our judgement was reasonable.

A variety of varieties

We recognise that studies will differ. It is helpful to identify three basic ways in which they differ: clinical diversity, methodological diversity and statistical heterogeneity. Heterogeneity and diversity are words that have pretty much the same meaning. We've used different words here as people often mean 'statistical heterogeneity' when they just say 'heterogeneity'.

Clinical diversity

We use the term 'clinical diversity' (sometimes called 'clinical heterogeneity') to describe clinical differences in the studies to do with the participants, interventions and outcomes. This covers such factors as

• study location and setting
• age, sex, diagnosis and disease severity of participants
• treatments people may be receiving at the start of a study
• dose or intensity of the intervention
• definitions of outcomes.

Methodological diversity

'Methodological diversity' (sometimes called 'methodological heterogeneity') covers differences between how the studies were executed, including such variables as

• a parallel group trial or a crossover trial
• randomization by clusters (for example, by family or by school) or by individuals, or by body parts (for example, eyes or different parts of the mouth)
• study quality (for example, the extent to which allocation to interventions was concealed, or whether outcomes were assessed blind to treatment allocation)
• analysis (for example, performing an intention-to-treat analysis compared with an 'as treated' analysis)

The distinction between some aspects of clinical and methodological diversity is not always clear-cut. For example, is the length of a study a feature of the intervention being evaluated, or of the outcome being assessed or of the study design? As long as we remember to assess it, it does not really matter how we classify it.

Before we go on to statistical heterogeneity, try to complete the activity based on your clinical knowledge of how the participants in your included trials may respond differently to the intervention, and your knowledge of the methodology of your included trials. It does not really matter which heading we put it under, as long as we consider it somewhere.

Do you think any of these differences are so great that studies should not be combined?

This is a difficult question to answer. To help you think about it, you can ask yourself the following questions:

• Could any of these differences make the treatment have the opposite effect to the one we want?
• Could any of these differences make the treatment work particularly well?

If you can think of situations in your review where this might be true, and there is good evidence to back up your suspicion, it might not be appropriate to pool all the studies together.

For example, if we look at aspirin as an intervention to prevent death from stroke, are there groups of patients who are more susceptible to the side effect of aspirin induced bleeding, which can actually cause death. In some groups this might outweigh any beneficial effect. Are there groups of patients who might particularly benefit, such as patients at high risk of stroke?

It's also important to realise that not every factor that influences how well a patient does in general (prognostic factors) will influence the size of the treatment effect. For example, the more severe a head injury is, the more likely you are to die. This doesn't necessarily mean that we should not combine studies in patients with different severities of head injury. The treatment may work equally well in any severity of head injury.

To summarise, an important decision when performing a systematic review is whether or not to combine studies. This decision needs to be made for each individual outcome of every comparison in your review. It is possible to perform a meta-analysis for some comparisons and not for others; depending on the individual studies you have found addressing this comparison. The decision to combine studies in a meta-analysis should be made based on the setting, participants, interventions and outcomes of the included trials being sensible to combine (i.e. little clinical diversity); and the methods used to perform the trial not varying in a way that is likely to overly influence the results (methodological diversity). To confirm or question your decision, you should consider statistical heterogeneity.

Statistical heterogeneity

Having decided that we wish to look at a group of similar studies together, we need some checks to see whether we have made the right judgement. We do this by looking at the estimates of treatment effect of the individual studies. As we are trying to use the meta-analysis to estimate a combined effect from a group of similar studies, we need to check that the effects found in the individual studies are similar enough that we are confident a combined estimate will be a meaningful description of the set of studies.

In doing this, we need to remember that the individual estimates of treatment effect will vary by chance, because of randomisation. So we expect some variation. What we need to know is whether there is more variation than we'd expect by chance alone. When this excessive variation occurs, we call it statistical heterogeneity, or just heterogeneity.