Publication Bias

The pdf file to your left explains the parts of a funnel plot and you should familiarise yourself with it before progressing any further.

In the printable version of this module, you will find these images at the end of the module.

In the absence of publication bias we might expect a symmetrical funnel plot.

Interpreting funnel plots

If publication bias is not present, you would expect your funnel plot to be roughly symmetrical, as in the example below:


As the studies become less precise (i.e. higher standard error), you would expect the results (given here as a log odds ratio) of the studies to be more variable, scattered to both sides of the more precise larger studies.

When you plot your studies onto a funnel plot, you may find it is not symmetrical and does not resemble an inverted funnel. This may be due to publication bias, however there are other factors leading to an asymmetrical plot.

The next funnel plot is from a review of prevention for chronic non-steroidal anti-inflammatory medication induced gastro-intestinal toxicity.

An asymmetrical funnel plot may be due to study factors other than publication bias As you can see it is not symmetrical, although this impression is mainly caused by one small study to the left of the most common effect. This may indicate publication bias, but there are other possible explanations. The small study may be of lesser quality, and poor quality studies, especially those failing to conceal allocation, often result in exaggerated treatment effect sizes. Or this small study may have been performed in a particularly high risk population where the effect is large. In looking at this plot, we can only report that there may be publication bias.

Look at the plot below from a review of Aversive Smoking for smoking cessation. The outcome is risk of quitting, so the larger the OR the better aversive smoking works.

Does this look symmetrical? At first look it appears that the smaller, less precise studies are all much more positive than the larger, more precise studies, and there are no smaller studies to the left (negative) side of the graph. This appears to be a good example of publication bias.

If however, we add the control event rates (quit rate in the control group) to the plot, the interpretation may be different.

The trials with the lowest control event rates demonstrate the most positive results. We could convince ourselves when looking at this that the pattern of greater effect with lower control event rates represented a true relationship, adverse smoking works better in those more addicted people less likely to give up anyway (i.e. without the experimental intervention). Or it could be publication bias. There are lots of possible explanations for this pattern. The point is that from the funnel plot it is impossible to know.

Another possible type of funnel plot is a hollow plot, like this one from a review of dieting to reduce body weight for controlling hypertension in adults.

Here there are some trials to the right of the point of no effect, indicating that dieting increases blood pressure (measured as the mean difference on a continuous scale) and some to the left, indicating that dieting reduces blood pressure. There are no trials around no difference. This is possibly publication bias of the type where significant studies (i.e. those showing the intervention is significantly beneficial and those showing the intervention is significantly harmful) are published or found systematically more than those showing no difference.

From these examples, we can see that a funnel plot is not a very reliable method of investigating publication bias, although it does give us some idea of whether our study results are scattered symmetrically around a central, more precise effect. Funnel plot asymmetry may be due to publication bias, but it may also result from clinical heterogeneity between studies (for example different control event rates) or methodological heterogeneity between studies (for example failure to conceal allocation).

Even if there is publication bias in a review, it may not result in an asymmetrical funnel plot, for example when the plot is hollow.

There are some statistical tests for detecting funnel plot asymmetry (Egger’s linear regression test and Begg’s rank correlation test) but these have low power and are rarely used in Cochrane reviews. If you would like to use them, you should discuss this with a statistician.