Risk of bias assessment of sequence generation: a study of 190 systematic reviews of trials

Session: 

Oral session: Investigating bias (2)

Date: 

Monday 17 September 2018 - 14:00 to 14:10

Location: 

Carrick 2

All authors in correct order:

Wuytack F1, Regan M2, Biesty L3, Meskell P4, Lutomski J5, O'Donnell M2, Treweek S6, Devane D3
1 School of Nursing & Midwifery, Trinity College Dublin, Ireland
2 School of Medicine, National University of Ireland Galway, Ireland
3 HRB-Trials Methodology Research Network/School of Nursing & Midwifery, National University of Ireland Galway, Ireland
4 Department of Nursing & Midwifery, University of Limerick, Ireland
5 Radboud Biobank, Radboud University Medical Center, The Netherlands
6 Health Services Research Unit, University of Aberdeen, UK
Presenting author and contact person

Presenting author:

Declan Devane

Contact person:

Declan Devane
Abstract text
Background: Systematic reviews help inform policy and healthcare decisions. Randomised trials are the optimal study design for evaluating the effectiveness of interventions. Random sequence generation is used to help minimise selection bias in the way participants are allocated to groups. We have observed that some reviewers judge randomised trials as at high or unclear risk of bias (ROB) for sequence generation, but the extent of this issue has not been well described.

Objectives: We evaluated the consistency in the ROB assessment for sequence generation for randomised trials in Cochrane and non-Cochrane reviews.

Methods: We retrieved Cochrane intervention reviews (1 Jan 2017 to 31 Mar 2017) from the Cochrane Database of Systematic Reviews. We also searched for systematic reviews in 10 general medical journals with the highest impact factors (1 Jan 2016 to 31 Mar 2017). We examined the proportion of reviews that rated the sequence generation domain as at high, low or unclear risk of selection bias. For reviews that had rated any randomised trials at high or unclear risk of bias, we examined the proportion that assessed the quality of evidence.

Results: We included 105 Cochrane and 85 non-Cochrane reviews. Of the 1352 randomised trials included across the Cochrane Reviews, 1% were rated at high and 46% at unclear ROB for sequence generation. For non-Cochrane reviews, these figures were 4% and 44%, respectively. The 65% of non-Cochrane reviews that had rated randomised trials at high ROB for sequence generation had not reported why. All Cochrane Reviews assessed the quality of evidence (GRADE). For the non-Cochrane reviews, only two-thirds had assessed the quality of evidence.

Conclusions: Systematic reviews of interventions frequently rate randomised trials at high or unclear ROB for sequence generation. Consistent ROB assessment for sequence generation contributes to minimise selection bias, and transparent quality of evidence assessment ensure that conclusions are linked to the data. In general, Cochrane reviews were more transparent than non-Cochrane reviews on ROB and quality of evidence judgements. Standardised reporting of trial characteristics should be strongly promoted to improve evaluation of evidence.

Patient or healthcare consumer involvement: No patients or consumers were directly involved in the conduct of this study.

Attachments: 

PDF icon Table 1 for Cochrane abstract, DDD.pdf

Relevance to patients and consumers: 

Systematic reviews are important to inform healthcare decisions for the benefit of end users. This study examined how reviewers assess bias in studies that they include in the review, specifically looking at bias related to how researchers decide how study participants are allocated to different groups. This study will improve the conduct of systematic reviews, which will be better able to inform healthcare decisions. This study did not have a patient/consumer partner.