Sensitivity analysis

The process of undertaking a systematic review and meta-analysis involves many decisions. Ideally, most of these are made while designing the protocol. It is usually necessary, however, to make further decisions in order to deal with the studies subsequently identified: few reviewers correctly anticipate all problems that arise. Many questions that arise during the review process will have obvious answers. Others will not have clear answers, and in some cases our decisions may change the whole conclusion of the review. The role of a sensitivity analysis is to determine whether the assumptions or decisions we have made do in fact have a major effect on the results of the review. You should present your investigations of the effect your assumptions had in the Results section of your review by detailing the range of treatment estimates and confidence intervals resulting from the various sensitivity analyses.

A sensitivity analysis addresses the question 'Are the findings robust to the method used to obtain them?' Sensitivity analyses involve comparing the results of two or more meta-analyses calculated using different assumptions. Here are a few examples of the sorts of things performed as sensitivity analyses in Cochrane reviews:

• If a study is of doubtful eligibility for the systematic review, then comparing meta-analyses excluding and including that study might be undertaken as a sensitivity analysis
• Results may be calculated using all studies and then excluding poorer quality studies
• Both fixed and random effects meta-analyses might be undertaken to assess the robustness of the results to the method used
• If a study appears to be an outlier (has results very different from the rest of the studies) then its influence on a meta-analysis might be assessed by excluding it
• Where missing information is 'imputed' (brought in from another source, perhaps by estimating it) then the effect of imputed numbers should be assessed through sensitivity analysis. This would normally take the form of re-analyzing the data using several alternative imputed values. This is frequently necessary when including cross-over trials, cluster randomized trials or change-from-baseline outcomes in meta-analysis.
• To determine whether a meta-analysis result is being heavily determined by a particular trial it might be repeated excluding that trial. The largest trial or the earliest trial could be driving the result, for example.

We take a more detailed look at one particular type of sensitivity analysis below when we address missing outcome data and intention to treat analyses.