Practicalities of sifting

A quick note on language

The information you're using for pre-screening and selection might not be written in your first language. When looking at studies in a language you cannot read, you are likely to need help. How much help you need depends on whether you can identify certain key words to tell you whether a study might be relevant. If not, you will probably need to find someone who is familiar with that language to help. They may just be able to read an abstract and tell you whether it is worth ordering the full article. With a full paper, you may just be able to ask someone to read it and tell you whether it is eligible, or you may need to get certain parts, or all, of the paper translated. You may know people who speak a variety of languages, or your review group may be able to help you find a translator if necessary.

Human error

Both stages of sifting are going to be done by humans. Humans make mistakes and people doing reviews are no different. When looking at a lot of studies we may simply miss some information and mistakenly include or exclude the study. We all have certain prejudices or biases which might make us more or less likely to choose a particular study. Part of the review process is to try to minimise these mistakes and biases.

An important point is that if you exclude a study in the initial sift, it will rarely get another chance to be included. So reviewers usually give studies the benefit of the doubt at this early stage, and go on to obtain the full report.

Should you collect outcome data at the same time as eligibility information?

It might seem that since you are reading these papers anyway for making eligibility decisions, why not collect outcome data at the same time? Why read all the papers again later?

There are two main arguments against this. The first is simple - if you decide to exclude a study, you will have extracted its data unnecessarily, and will have wasted time and effort. We mentioned the second reason earlier on - you should try to avoid letting the results of studies sway your decision. If you look too hard at the results too soon, you might find it difficult not to be swayed.

Reducing mistakes

How can we reduce mistakes? A common sense way to try and do this is to have more than one person make the decisions. Then, if one person makes a mistake, there will be one or more others who might get it right. In one review, pairs of reviewers independently sifted about 11,000 records. It took the reviewers between 11 and 28 hours to go through all these records. The reviewers concluded that having two people check each record was worthwhile. The reference for this study is listed at the start of the module under 'Other relevant material'. Therefore, assuming that your review has many fewer records than that large review, the time needed for this part of your review might be less than you had thought. So you should think about having more than one of you do this stage.

If you can't find one or more other people willing to commit the necessary amount of time, there are a couple of compromises

• have a second person look at a sample of the records
• have two reviewers each look at separate sets of records, and overlap the sets they look at so that they both do some of the records (for example one looks at years 1966-1988 and the other looks at 1986-2000)

If you find, through either of these ways of working, that you disagree on quite a few records, you may need to look again at the criteria you have set for the sifting of the records of studies. There are no rules about the level of disagreement above which you need to go back and redo the work. It should be low, as mistakes here might mean that you leave out relevant studies, which are unlikely ever to be looked at again.

If you do think you need to go back and recheck this sifting process, the most obvious option is to resift all the records, with two or more reviewers looking at all of them. Another option is for one of the reviewers to sift through all the records they did not look at first time.

For the second stage of the sifting, there will be fewer reports and it seems common sense that at least two reviewers look at each of them. Of course, there may be occasions when you think the decisions are likely to be so straightforward that this isn't necessary. Whatever you decide to do, report it in the Methods section of your review.