Choosing studies likely to give a valid answer to the question

These modules are mainly concerned with systematic reviews of health care interventions - questions of the type 'Does intervention A have different effects to intervention B in this health problem?' The Cochrane Collaboration does not currently do reviews of diagnostic test accuracy, or other types of health questions such as prevalence, prognosis or genetic predisposition.

Examples of questions about interventions, from published reviews, are:

• does the application of compression bandages or stockings aid venous ulcer healing?
• what are the effects of topical treatments applied during pregnancy on the later development of stretch marks?

These questions are about comparing different interventions and measuring their effects. This means there should be comparison groups, where one group receives one intervention and the other group receives the alternative.

An important issue in designing studies like this is the generation of the comparison groups. We want the two groups to be identical in every respect, except for the different interventions they get. If the groups are not identical, we are often not sure whether differences in the outcomes between the two groups are due to these differences in the groups, or due to the interventions we wanted to study.

For example, if we wanted to compare the effect of compression bandages for venous ulcers with doing nothing, we would want comparison groups which had similar ages of patients, similar proportions of people with diabetes, similar proportions of smokers, etc. In fact, we would want anything that might affect wound healing to be equal in the two groups, and the only difference to be that one group gets compression bandages and the other doesn't.

We could try to make sure of this by 'matching' patients - for every person in the group getting compression bandages, we could try to find another person with similar age, same sex, diabetes and smoking status, and put them in the group not getting compression bandages.

As well as taking a lot of effort, the problem is that there may be factors influencing wound healing that we haven't thought about, or don't know about. We can't match for these. This is where the power of randomisation lies. If each person coming into the study has an equal chance of going into either group, over the long run all factors, known and unknown, will be equally distributed.

So, randomisation should produce comparable groups, and the groups are more likely to be comparable the more people are randomised. This means we are more certain about concluding that differences in the outcome are due to the treatment. In short, the studies are less likely to be biased, and we are more likely to believe the results. Studies that randomise participants to groups are called randomised controlled trials (RCTs).

There are other practical reasons for focussing on randomised controlled trials. These reflect some of the methodological work done in this area, and some of the work done by the Cochrane Collaboration over the last few years:

• a considerable amount of work has gone into making randomised trials easier to find, so that we don't find a non-representative, or biased, group of studies
• we are beginning to understand how to tell an unbiased trial from a biased one

The same amount of background work has not been put into the location and utilisation of other study designs. For this reason, concerns about interpreting results will be greater if we choose study designs other than randomised controlled trials. Most Cochrane reviews therefore use the presence of randomisation as a minimum quality criterion when deciding which studies to include and, therefore, only include randomised trials.

What if there aren't any trials?

Randomised controlled trials are more common in some areas of health care than others, for example where the interventions are drugs. What do we do if we have an important question, but there are no randomised controlled trials addressing the question?

Well, there's absolutely nothing 'wrong' with a systematic review that has been done to a high standard, but finds no studies. In fact, these reviews are very useful because they highlight important gaps in our knowledge. Research funders are increasingly looking at the results of systematic reviews to help them decide what studies to commission. So a review that finds no studies can stimulate new research that will be able to answer the question.

See what you think of a review

It's a good idea to have a look at a systematic review for yourself. Find a recently published systematic review, for example by looking at The Cochrane Library, or searching PubMed (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi) using the term 'systematic review'. Read the review, and ask yourself how well you think it summarises the evidence.

There's no need to carry out a formal appraisal of it; just think about how relevant it is, and how accurate you think it is. Can you think of any ways in which it could have been done better? Make some notes and keep them somewhere, so that you can come back to them at the end of these modules and think again about the quality of this review, and how you could do it better.