Assessing the certainty of evidence in overviews of reviews: current practice and expert perspectives

ID: 

273

Session: 

Poster session 2

Date: 

Monday 17 September 2018 - 12:30 to 14:00

All authors in correct order:

Brennan S1, Middleton P2, Akl E3, Pollock A4, Reid J5, McKenzie J5
1 Cochrane Australia, Monash University, Australia
2 SAHMRI; Robinson Research Institute, The University of Adelaide, Australia
3 Centre for Systematic Reviews on Health Policy and Systems Research, American University of Beirut, Lebanon
4 Nursing, Midwifery and Allied Health Professions Research Unit, Glasgow Caledonian University, UK
5 School of Public Health and Preventive Medicine, Monash University, Australia
Presenting author and contact person

Presenting author:

Sue Brennan

Contact person:

Sue Brennan
Joanne McKenzie
Abstract text
Background:
Proliferation of systematic reviews (SRs) and escalating demand for accessible evidence have driven interest in overviews of reviews. Unique challenges arise in synthesising review evidence, for which guidance is yet to be agreed. These challenges have ramifications for assessing the certainty of evidence using GRADE or other approaches. Authors must decide, for example, how to 1) account for limitations of the included SRs, 2) deal with overlapping data (e.g. where the same primary studies contribute to meta-analyses in multiple reviews), and 3) handle missing or discordant data needed to assess certainty (e.g. conflicting risk of bias assessments).

Objectives:
To inform guidance for assessing the certainty of evidence in overviews by examining current practice and expert perspectives.

Methods:
We undertook a systematic review of methods used to assess the certainty of the evidence in all eligible overviews (i.e. those reporting formal or informal assessment of certainty) indexed in MEDLINE (January 2015 to March 2017). We carried out interviews followed by a workshop (Global Evidence Summit, 2017) with GRADE, overview and guideline methodologists about their views on assessing certainty of evidence in overviews. Methodologists were presented with scenarios common in overviews (e.g. two overlapping meta-analyses from SRs of different quality and currency), and asked to describe and comment on approaches to rating certainty.

Results:
From 1707 citations, 41 of 131 overviews were included; 19 extracted or reported GRADE assessments (most without modification for overviews), 16 used another tool and six used ad hoc criteria. Participants in interviews and the workshop agreed that both certainty of the primary evidence and the SR process are important, but held diverse views on whether to integrate SR limitations in the rating of certainty. The overview purpose and methods dictate approaches to assessing certainty if data are overlapping, missing or discordant. For example, in dealing with overlap, the approach differs if selecting one meta-analysis to GRADE rather than conducting a new meta-analysis.

Conclusions:
Our results provide principles for assessing the certainty of evidence in an overview. However, different approaches may be warranted for incorporating limitations of the SR when assessing certainty of the evidence.

Patient or healthcare consumer involvement:
None.

Relevance to patients and consumers: 

Overviews of reviews have been identified as an important review format for making evidence accessible to healthcare consumers. However, careful interpretation of overview findings, and the certainty of those findings, is needed to ensure reliable information for decision-making. This research aims to inform guidance for assessing certainty of the evidence in overviews.