Combining studies

Which method should I use in my review?

We have talked about three methods of combining the results of trials included in a meta-analysis, the fixed effect method (the Mantel-Haenszel approach, which is more specific to dichotomous data and weights studies in a slightly different way), the Peto method which is useful for rare events, and the random effects model, which assumes that all studies are estimating their own true effect, and these effects are normally distributed. The analysis program within RevMan allows us to choose which of these models we want to use in our meta-analysis, and the results of our review will be slightly different depending on which method we use. The table below summarises the summary effect and confidence intervals resulting from selecting each of these methods in the daycare example we used earlier.

Method RR (95% CI) OR (95% CI) RD (95% CI)
Mantel-Haenszel 0.64
(0.49,0.82)
0.47
(0.31, 0.73)
-0.14
(-0.23, -0.06)
Peto Method 0.47
(0.30, 0.72)
Random effect inverse-variance 0.64
(0.50, 0.82)
0.47
(0.31, 0.72)
– 0.13
(-0.25, -0.01)

As you can see from the table, there is little difference in the results regardless of which method you choose, and the conclusions of your review would certainly not change. So why do we devote so much energy to selecting the summary statistic and devising methods with varying assumptions? There are some cases or circumstances where one method performs better than the others, and if any of these circumstances fit your review you may need to think carefully about which statistic you use.

Some general points about the performance of the various statistics

  1. The Mantel-Haenszel methods have been shown to be more reliable when there are not many data (small trials and not many of them). This is why they have been selected as the principle method of meta-analysis in the Cochrane Collaboration. This method (which can be used for OR, RR and RD) is the most appropriate for many Cochrane reviews, and many Cochrane review groups use it as standard. But it should not be used in reviews with sparse data, where lots of trials have zero events in treatment or control groups or both. The choice between OR, RR or RD should be based on the information covered in Module 11.
  2. The Peto method performs well with sparse data and is then the best choice, but when events are common there is usually no preference to use it over the other methods. It is not a good idea to use the Peto method when the treatment effect is very large, as the result may be misleading. This method is also unsuitable if there are large imbalances in the size of groups within trials.
  3. A random effects model may be better when there is statistical heterogeneity between the studies in your review (we will discuss this further in Module 13 on Heterogeneity).

Summary

From this module and the one preceding it we can see that there are many choices the reviewer has to make about the way they analyse their dichotomous data in a review. Firstly decisions about which within trial statistic to use (OR, RR or RD) need to be made, and then the method of combining the trial data, or meta-analysis, needs to be chosen. While there is not consensus about which is the best approach, and you will need to check your review group policy, following the principles set out here should help you make your decision. If you are concerned that your review falls into one of the categories where there are special considerations (rare events with zero cells, very large treatment effects, or large variation in the control event rates between the trials included in your review) you may want to seek the advice of a statistician, and your review group can help you with this.