Things you can do with diversity and heterogeneity

If you identify or suspect that important diversity or heterogeneity is present in your review, there a several options open to you. Don't forget that one option is that of not performing a meta-analysis. An unwise meta-analysis can lead to highly misleading conclusions. If you have clinical, methodological or statistical heterogeneity it may be better to present your review as a systematic review using a more qualitative approach to combining results, or to combine studies only for some comparisons or outcomes. Studies can always be entered into RevMan and presented on a forest plot with their individual effect sizes and no combined effect. This gives an overall picture of the evidence.

Another alternative if there are subgroups of patients who are likely to respond very differently is to undertake separate reviews. For example, there are separate Cochrane reviews of influenza vaccines in healthy adults, people with cystic fibrosis, people with asthma and people with chronic obstructive pulmonary disease. This sort of decision should, of course, be made at the question formulation stage.

In the remainder of this module we take a brief look at three options for investigating or incorporating heterogeneity in a review:

• using a different statistical model for combining studies, called a random effects meta-analysis
• investigate heterogeneity by splitting the studies into subgroups and looking at the forest plot
• investigating heterogeneity using meta-regression

Fixed and random effects meta-analysis

We briefly discussed the 'fixed effect' and 'random effects' options for meta-analysis available in RevMan in Module 12.

Fixed and random effects meta-analyses sometimes give you similar results and sometimes give you results that differ. We'll explain what the technical difference is, then explain what a difference in the results implies.

Fixed effect meta-analysis

Methods of fixed effect meta-analysis are based on the mathematical assumption that a single common (or 'fixed') effect underlies every study in the meta-analysis. In other words, if we were doing a meta-analysis of odds ratios, we would assume that every study is estimating the same odds ratio. Under this assumption, if every study were infinitely large, every study would yield an identical result. This is the same as assuming there is no (statistical) heterogeneity among the studies.

Random effects meta-analysis

A random effects analysis makes the assumption that individual studies are estimating different treatment effects. In order to make some sense of the different effects we assume they have a distribution with some central value and some degree of variability. The idea of a random effects meta-analysis is to learn about this distribution of effects across different studies. By convention (but unfortunately) most interest is focused on the central value, or mean, of the distribution of effects. This is what the statistical part of RevMan presents when you select a random effects meta-analysis. It is also important to know the variability of effects.

What are the important differences between fixed and random effects and which one should I choose?

The first point is that you should analyse your review in both ways (i.e. select first one option then the other in RevMan) and see how the results vary. If fixed effect and random effect meta-analyses give identical results then it is unlikely that there is important statistical heterogeneity, and it doesn't matter which one you present. If however, your results vary a little, you will need to decide which is the better method on which to base your conclusions (usually it will be best to select the most conservative option).

There is a great deal of debate between statisticians about whether it is better to use a fixed or random effect meta-analysis. The debate is not about whether the underlying assumption of a fixed effect is likely (clearly it isn't) but more about which is the better trade off, stable robust techniques with an unlikely underlying assumption (fixed effect) or less stable, sometimes unpredictable techniques based on a somewhat more likely assumption (random effects).

Sometimes the point estimate of the treatment effect differs between fixed and random effects because of publication or quality related bias. This may indicate that careful investigations are required, perhaps with expert methodological input. If this is the case in your review you should check with your review group.

Keeping it all in context

It's important to remember that whatever statistical model you choose, you have to be confident that clinical and methodological diversity is not so great that we should not be combining studies at all. This is a judgement, based on evidence, about how we think the treatment effect might vary in different circumstances. This judgement is a common source of disagreement about the results of meta-analyses. Make sure you spend enough time considering this judgement in some depth before you worry too much about which statistical model you choose.