Assessing imprecision in Cochrane systematic reviews: a comparison of GRADE and Trial Sequential Analysis

Session: 

Oral session: Statistical methods (1)

Date: 

Monday 17 September 2018 - 11:20 to 11:30

Location: 

Fintry Auditorium

All authors in correct order:

Castellini G1, Bruschettini M2, Gianola S3, Gluud C4, Moja L1
1 Department of Biomedical Sciences for Health, University of Milan, Milan, Unit of Clinical Epidemiology, IRCCS Galeazzi Orthopedic Institute, Milan, Italy
2 Department of Paediatrics, Skane University Hospital, Lund University, Department of Research & Development, Section for HTA analysis, Lund, Sweden
3 Unit of Clinical Epidemiology, IRCCS Galeazzi Orthopedic Institute, Milan, Center of Biostatistics for Clinical Epidemiology, Department of Health Sciences, University of Milan-Bicocca, Monza, Italy
4 The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen, Denmark
Presenting author and contact person

Presenting author:

Christian Gluud

Contact person:

Greta Castellini
Abstract text
Background: The evaluation of imprecision is a key dimension of the 'Risk of bias' assessment. GRADE gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use of it. Trial Sequential Analysis (TSA) has been advocated for a more reliable assessment of imprecision.

Objectives: To evaluate reporting of and adherence to GRADE, and to compare the assessment of imprecision of intervention effects assessed by GRADE and Trial Sequential Analysis (TSA) in Cochrane systematic reviews.

Methods: A cross-sectional study: we included 100 Cochrane Reviews irrespective of the type of intervention, provided they considered a key dichotomous outcome meta-analysed and assessed by GRADE. We assessed the Methods and Results sections of each review for adequacy of evaluation of imprecision. We reanalysed imprecision following the GRADE Handbook and the TSA Manual.

Results: Overall, only 12.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of confidence intervals (CI) and the optimal information size (Table 1). Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was reanalysed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors’ assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% CI 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors on GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between GRADE reanalysed following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79; Table 2).

Conclusions: In a sample of Cochrane Reviews, methods for assessing imprecision were rarely reported. Quality of evidence was often downgraded due to imprecision. Strict adherence to GRADE Handbook guidelines led to more frequent downgrading because of imprecision, similar to that obtained by TSA.

Patient or healthcare consumer involvement: Our project is a step towards the need for clinicians and researchers to find the most accurate method to be confident in the clinical findings so that reliable and precise results can be transferred safely to clinical practice.

Attachments: 

File Table 1.tif
File Table 2.tif

Relevance to patients and consumers: 

Stakeholders should be cautious and critical when reading a systematic review that does not present an analysis of the overall results’ imprecision. Determining the extent of misleading conclusions and inflated results is paramount as users of systematic reviews and clinical practice guideline developers tend to depend on their results when framing recommendations necessary to patients and healthcare consumers. Our project is a step towards the need for clinicians and researchers to find the most accurate method to be confident in the clinical findings so that reliable and precise results can be transfered safely to clinical practice.