What is publication bias?

Systematic reviews aim to find and assess for inclusion all high quality studies addressing the question of the review. But finding all studies is not always possible and we have no way of knowing what we have missed. Does it matter if we miss some of the studies? It will certainly matter if the studies we have failed to find differ systematically from the ones we have found. Not only will we have less information available than if we had all the studies, but we might come up with the wrong answer if the studies we have are unrepresentative of all those that have been done.

We have good reason to be concerned about this, as many researchers have shown that those studies with significant, positive, results are easier to find than those with non-significant or 'negative' results. The subsequent over-representation of positive studies in systematic reviews may mean that our reviews are biased toward a positive result.

Publication bias is just one type of a group of biases termed reporting bias. We have quite a lot of evidence that these biases exist, so it is fair to assume that most systematic reviews will be subject to reporting bias to some extent.

Publication bias and other related biases can be summarised as statistically significant, 'positive' results being:

• more likely to be published (publication bias)
• more likely to be published rapidly (time lag bias)
• more likely to be published in English (language bias)
• more likely to be published more than once (multiple publication bias)
• more likely to be cited by others (citation bias)

All of these reporting biases make positive studies easier to find than those with non-significant results, something that we can try to minimise by extensive searching.

Managing publication bias

If we accept that your review will almost certainly be subject to publication bias to some extent, we are left with the problem of estimating how big a problem it is in your review, and what to do about it. There are several methods for getting an idea about how much of a problem this may be, and the method available in RevMan is the funnel plot. This is available in RevMan as a tool for reviewers, but is not on The Cochrane Library. This means you should use the funnel plot option to investigate the presence of publication bias in your review and then discuss this in the Discussion section of the text of your review. If you suspect there may be a problem in your review, you need to bear this in mind when making your conclusions and recommendations. The likeliest scenario is that the results of your review are biased to the positive.