Indirect comparisons

We talked a little about indirect comparisons when we considered subgroup analysis in the module on heterogeneity. We frequently wish to compare interventions across studies. Here are two examples.

• In Module 13 we thought about a collection of trials comparing training of systematic reviewers with no training of systematic reviewers. Some trials used self-directed learning and others used face-to-face training. None compared the two types of training, although we really want to know which one works best.
• A Cochrane review of pharmacological interventions for dyspepsia included many comparisons between different drugs and between drugs and placebo. Among them were comparisons of histamine H2 antagonists (H2RAs) versus placebo, comparisons of prokinetics versus placebo and comparisons of H2RAs versus prokinetics. There were substantially more placebo-controlled comparisons, yet an important clinical question is whether H2RAs are more effective than prokinetics. Could the placebo-controlled studies reliably tell the reviewers more about this comparison?

It is clear that the best way to compare treatments is to seek direct comparisons in randomized trials. Sometimes there aren't enough of these studies to draw reliable conclusions, and sometimes there aren't any such studies at all. Here we address some of the issues involved in making indirect comparisons across different studies.

The first question is whether we should compare the treatment arms in the different studies, or compare the treatment effect estimates from the different studies.

Suppose we had a single trial of an H2RA versus placebo and a single trial of a prokinetic versus placebo. Can we reliably compare the patients given the H2RA in one trial with the patients given the prokinetic in the other trial? The answer is clearly no. These groups are unlikely to be comparable, because there are likely to be differences in the case-mix, outcome assessments and other aspects of trial design between the trials. Although the groups have been generated by randomisation, the randomisation acted within each trial (H2RA versus placebo, prokinetics versus placebo). Participants were not randomized to be in one trial or the other, i.e. there was no randomisation to prokinetics versus H2RA.

A more reasonable approach is to exploit the randomization within each study, and compare treatment effect estimates. This does not make the comparison randomized, but makes it considerably more reliable since we now relate the effects of the two active drugs to a common reference. For example, if we have one trial comparing Drug A to placebo, and one comparing Drug B to placebo, we could compare the treatment effects measured in the two studies as an indirect way of comparing Drug A to Drug B.

We have determined that indirect comparisons should exploit the comparisons within randomized trials. But when is it reasonable to make indirect comparisons like that suggested above? We need to ask ourselves, 'are the studies sufficiently similar to provide a meaningful result?'.

Remember, this comparison between the results of two studies is still a non-randomized comparison. The problem with making indirect comparisons is that there are often systematic differences between the types of trials addressing one intervention and the types of trials addressing the other. For example, trials of self-directed learning for systematic reviewers may be undertaken in resource-limited countries, and trials of face-to-face training in resource-rich countries. There may be reasons, other than the approach to learning, for differences in the effectiveness of training in these two situations.

Methods are available for undertaking indirect comparisons if it is reasonable to do so. Meta-regression is one approach. Another approach is to formally compare the estimates and confidence intervals for the direct comparisons. Neither of these methods is available in RevMan.

If we are going to use indirect comparisons, we need to be very cautious about their interpretation.

Summary

We have covered several issues in this module.

Sensitivity analyses are a way of investigating the importance of some of the assumptions and decisions we make during a systematic review.

We have briefly covered some types of data encountered during reviews, and looked at the difficult issue of the value of indirect comparisons.

In the next modules, we start to look at some important issues in the interpretation of the data we have collected.