The ROB-ME (Risk Of Bias due to Missing Evidence) tool: a new tool for assessing reporting biases in evidence syntheses

Workshop category: 

  • Investigating bias
Date and Location

Date: 

Monday 17 September 2018 - 14:00 to 15:30

Location: 

Harris 2
Contact persons and facilitators

Contact person:

Matthew Page

Facilitators:

Page M1, Sterne J2, Higgins J2
1 Monash University, Australia
2 University of Bristol, United Kingdom

Acknowledgements:

Target audience

Target audience: 

Review authors, Researchers, Trainers, Editors

Level of difficulty: 

Basic
Type of workshop

Type of workshop : 

Training
Abstract

Abstract:

Background:
The credibility of evidence syntheses can be compromised by reporting biases. These include 'publication bias', when the probability that a study is published depends on its findings, and 'outcome reporting bias', when findings that are statistically non-significant are less likely to be reported. Reporting biases imply bias in evidence syntheses because the missing results differ systematically from the available results. Existing tools for assessing the risk of reporting biases in evidence syntheses are limited in terms of their scope, guidance for reaching risk of bias judgements and measurement properties.

Objectives:
To introduce ROB-ME, a comprehensive new tool for assessing the risk of reporting biases in evidence syntheses, and to provide participants with the opportunity to apply ROB-ME.

Description:
The workshop will be split into two parts.

1) Introduction to ROB-ME:
We will provide a brief overview of the development and structure of ROB-ME. Key components of the tool include:

- preliminary consideration of which outcomes were pre-specified (e.g. in trial registries, protocols) in each study identified, and whether results are fully available, partially available or not available for outcomes of interest;
- specification of a particular synthesis to be assessed for risk of bias;
- use of signalling questions to inform risk of bias judgements (such as how many studies are definitely missing from the synthesis because results are not available or are partially available, and whether other studies are likely to be missing because of non-comprehensive searching);
- algorithms to map responses to signalling questions to judgements about risk of bias.

2) Applying ROB-ME:
Participants will apply ROB-ME to an example review in small groups. Each group will assess the example review for a particular component of the ROB-ME tool. In a keynote session we will discuss the results of each group's assessment and any issues that arose during the assessment process.

The workshop will conclude with a facilitated, structured discussion focusing on the implications of using the tool alongside other risk of bias tools (e.g. RoB 2.0), and further development needs for guidance and software.

Relevance to patients and consumers: 

Failing to consider reporting biases can result in the uptake of ineffective and harmful interventions. The recent Tamiflu saga illustrates this problem. Only after gaining access to withheld data from unpublished trials was a team of Cochrane reviewers able to determine that Tamiflu, a drug which the UK government spent £0.5bn on to prepare for flu pandemics, was ineffective. The new ROB-ME tool will enable those who perform evidence syntheses to provide more trustworthy conclusions, and thus has the potential to impact patients and the public through improved provision of care.