Reducing bias

All of us have prejudices that might affect our judgements about whether studies should be included or excluded. Experts may have pre-formed opinions which might affect their assessments of the relevance and validity of studies. On the other hand, it's difficult to make judgements if you know nothing at all about a topic. Other people might have opinions about the value of research published in particular journals, or research carried out in particular institutions.

One way to minimise the effect of these personal biases is to have reviewers of different backgrounds making judgements about studies, for example an expert and a non-expert. For this to work, however, both reviewers need to be willing to accept that they may have biases and to listen to the other reviewer's views!

One suggestion for reducing bias is that we should remove as much identifying information from the papers as possible, such as the name of the journal, authors, institutions, etc. A randomised trial has been done to assess this approach (you can find the reference as Berlin 1997 in the Cochrane Handbook for Systematic Reviews of Interventions, section 7.2.4). The trialists repeated several meta-analyses, once with reviewers who saw the unblinded papers of the included studies, and once with reviewers blinded to many of the identifying details in the papers. It took many hours to remove as much identifying detail as possible, and it appeared to make little difference to the overall estimate of effect for the meta-analysis. So, unless someone refutes these results, it looks as though we don’t normally need to ‘blind’ the papers, unless we think there are important reasons why not blinding the papers would be a problem for a particular review.

Another interpretation of the above study is that we are just not very good at concealing the identity of the papers. For instance, experts will often have read the paper before and may recognise the text.

Getting hold of extra information

You will often find that, even when you have the full report of a study, you don't have all the information you need. Maybe it doesn't tell you how the study was randomised, or maybe it doesn't tell you clearly who the participants were. This may happen later on as well, when you need outcome data.

You may be able to make assumptions about a study that everyone would agree are reasonable. For instance, if a report of a study tells you that it recruited women aged over 60, it is reasonable to assume that the women were all post-menopausal. There isn't really a need to ask the authors to clarify this. If you cannot make assumptions like this, you may need to create categories for features of trials called 'unclear' - it was 'unclear' whether the patients were masked to the treatment, etc.

The other way to deal with the situation is to try to get hold of more information. To do this you either need to find other reports of the same study, which contain more information, or try to contact the people who conducted the study. Make sure you've looked carefully for other reports - they will probably have come up in your search anyway, but you could do a quick extra search for other publications by the same author

Resolving disagreements

Just as surely as we all make mistakes, people will disagree about the way they see things. So, when you have two people making decisions about including and excluding studies, there will be times when you come to a different decision about a study.

Some people like to measure and report how often this occurs as part of their review, to give readers an idea of how difficult the decisions were. The Cochrane Collaboration doesn't insist that this is done.

To resolve differences, you first need to work out why you came to different judgements. It may just be that one person missed a vital sentence. It may be that there wasn’t enough information – in this case you might need to put that study aside and wait until you’ve got the extra information you need. Or maybe one reviewer has special knowledge about a study that isn’t in the report – they may even have been one of the authors. If you really can’t agree, ask another person to help you resolve this. You might choose another content expert or a methodologist, depending on the area of your disagreement. 

It is important to plan how you are going to resolve disagreements early on in planning your review and to ensure your review team are happy with the planned process. This will help resolve any possible later disagreement.

Getting your work into RevMan

Once you’ve decided which studies are included, excluded, awaiting classification (the ones where you needed more information), or ongoing (not yet completed), you’ll need to put them in the appropriate category in RevMan. This is explained in RevMan ‘Help’ under the heading ‘Studies and References’.