Traps for the unwary

There are several common mistakes frequently made by reviewers:

i. Evidence of no effect versus no evidence of effect

Often, when there is inconclusive evidence we confuse 'no evidence of effect' with 'evidence of no effect'. These two statements are not the same thing.

Often when there is insufficient evidence, a summary estimate will have wide confidence intervals that will include both the null and the possibility of quite big effects (either helpful or harmful). As an example, let's consider a summary relative risk of 1.02 with 95% confidence intervals that extend from 0.74 to 1.40. In this situation, there is no (or insufficient) evidence of effect, We can't say with any confidence that there is no effect, and it could be in either direction.

What we need is more evidence to increase our capability (or power) of detecting an important effect. If we wrongly concluded from the available evidence that the intervention had no effect when it really does, this would be referred to as Type II error (i.e. false negative, or missing an effect that is actually there). For this example it may be better to conclude along the lines of "the results of this review are consistent with the intervention being effective or not effective and further research is needed".

As a second example take the identical relative risk, but with much more precise 95% confidence intervals from 0.98 to 1.06. This time we can say there is evidence that, if there is an effect, it is small and might be too small to be important to users of health care. Statements like this will depend on what size of effect is clinically important. A 2% change in risk might be clinically important in some situations, but not others.

Irrespective of where the boundaries of the 95% confidence interval lie, we need to remember there is a 5% possibility that the true summary effect lies outside of these boundaries, and our result may be incorrect. If we wanted to be more certain, we could calculate the 99% confidence intervals around the summary estimate.

ii. Evidence-conclusion mismatches

Another error is a tendency to try and draw conclusions beyond the available evidence. It is very important that every statement you make in your review conclusions is backed up by your results, and all important results are dealt with in your conclusions. A trend towards an effect cannot be interpreted as a positive effect. It is important that you don't become guilty of reporting bias by only including the outcomes of review for which there was a significant effect in the conclusion and abstract of your review. As a minimum, results for the important outcomes should be reported in the conclusions and abstract.

iii. "Further research is needed"

A third common mistake is to be non-specific about the need for further research. A statement like "further research is needed" is not particularly helpful. The research that is needed should be described.

While it is hard to be prescriptive about how to draw inferences and conclusions from a review, try and practice by using these common mistakes as a form of check-list against which to try and improve conclusions from your own review. In doing a review you also need to remind yourself that no matter how well it is done, the review itself cannot make a health care decision, it can only assist in the process. Furthermore, once someone has used a review to make a decision there is no guarantee that they will derive the predicted benefits or avoid the potential harms. There never can be certainty in how individuals or groups of individuals will respond to an intervention. All we can hope to do is provide them with an objective summary of the best available evidence to inform their own treatment choices.

Try it for yourself

If you have access to The Cochrane Library, choose a review, read the results, without looking at the conclusions, and then try to write some conclusions. Then compare your conclusions with those of the authors. Are they the same? If not, why do you think they differ?