Diversity and heterogeneity

Identifying statistical heterogeneity

You can determine the presence of statistical heterogeneity in two main ways:

To identify heterogeneity you can visually assess the forest plot or perform a statistical test
  • by looking at a forest plot to see how well the confidence intervals overlap. If the confidence intervals of two studies don’t overlap at all, there is likely to be more variation between the study results than what you would expect by chance (unless there are lots of studies), and you should suspect heterogeneity
  • by performing a statistical test, known as a (“chi-squared”) test.

The result of this statistical test appears at the bottom of each meta-analysis within the statistical part of RevMan.

The result of the test is (i) a ‘chi-squared’ statistic (ii) a number called the degrees of freedom (which is usually one less than the number of studies, but can be less if some of the studies have no events, as in the example above) and (iii) a ‘p-value’ obtained by referring the first two numbers to statistical tables. A small p-value is often used to indicate evidence of heterogeneity (this p-value appears in RevMan and the Internet version of The Cochrane Library. It is not yet available in the CD-ROM version of The Cochrane Library).

As it applies to Cochrane reviews, this test is of somewhat limited value. This is because most meta-analyses in Cochrane reviews have very few studies in them. When there are few studies, the test is not very good at detecting heterogeneity if it is present (it has ‘low power’). For this reason, a p-value of less than 0.10 is often used to indicate heterogeneity rather than the conventional cutpoint of p = 0.05.

Conversely, if there are a lot of studies in a meta-analysis, the test can be too good at detecting heterogeneity. Since we have established that heterogeneity is almost certain to be present as studies are rarely identical, the test will detect significant heterogeneity even if it is clinically trivial (the test has too much power). But the basic problem is that the test does not answer a useful question. It asks the question ‘Is there heterogeneity?’ whereas we want to know ‘How much heterogeneity is there?’

A useful way to identify heterogeneity without having to use statistical tables to look up p-values is to compare the chi-square statistic with its degrees of freedom. If the statistic is bigger than its degrees of freedom then there is evidence of heterogeneity. A visual inspection of the confidence intervals will help get an idea of the amount of statistical heterogeneity, and guide you to think about whether it is reasonable to combine the results of these studies.