What do we mean by quality of trials?

Write a list of all the factors that come to mind when you think about the quality of a randomised trial.

You've probably come up with quite a long list. Some things on the list will be to do with the design of the study (blinding, sample size), and some to do with the way it was reported (the presentation of tables, who the authors are).

This long list in part explains why there have been many different approaches to measuring quality. So which bits are we interested in for this part of a Cochrane review?

Randomised trials, like systematic reviews, are trying to measure some 'truth' about an average effect of an intervention in a group of participants. When we talk about trial quality, we're usually talking about how well we think the study has measured this 'true' effect - this aspect of quality is also called validity.

But since no-one knows what the 'true' effect is, there's an element of guesswork and judgement in knowing which factors are most likely to affect how the study measures it. This is another reason for the many different approaches people have taken.

Measuring validity - measuring what?

The approach taken within the Cochrane Collaboration is first of all to think of factors which might lead to bias in studies. Then we take into account any studies that can tell us which of these are more important. Once we've decided which factors we're interested in, we collect information about them from the study reports.

Now read the first few sections of chapter 8 of the Cochrane Handbook for Systematic Reviews of Interventions, for an explanation of the sources of bias and which seem to be more important.

In summary, empirical evidence suggests that

• Allocation concealment is very important in protecting against bias, so we should look for information on what was done about this and report what we find
• Blinding of the interventions and the outcome assessment may be important
• Loss to follow-up of study participants may be important

Some people choose to report information on all these, others choose to collect more items, and some use scales to get overall quality scores. These scores seem attractive in comparing studies with each other, but there are problems with them. Since we have no ‘gold standard’ to check them against, it is difficult to tell how well the scales measure quality. Some scales contain items that are not really to do with validity, and there are worries that adding things up to produce an overall score makes assumptions about the relative importance of different items. For these reasons, the use of quality scales is NOT recommended for Cochrane reviews.

This is a hotly debated topic within the Cochrane Collaboration, and your review group may have a policy on how to assess quality. You could also look for Cochrane reviews that have addressed similar questions to see what those reviewers did. If you do plan to use a scale to measure quality, then you should choose one that has been tested for its reliability and validity, as with any scale

Practical issues, minimising mistakes and bias

As with eligibility decisions, you’ll want to record the information you collect on quality somewhere. Most people record it on their data collection form (we will cover this when we return to Module 7) and put it into the table of ‘Characteristics of included studies’ under the ‘Methods’ field in RevMan later.

The issues in minimising mistakes and bias while assessing study quality are the same as when we considered these in selecting studies, and you should report how you intend to assess quality, and how you did it:

• how many reviewers will do this?
• what are their backgrounds (for example, are they authors on any of the trials)?
• will you try and blind the reviewers to details of the papers, such as journals or authors?
• will you formally assess the reliability of the process