The ROB-ME (Risk Of Bias due to Missing Evidence) tool: a new tool for assessing reporting biases in evidence syntheses

Session: 

Oral session: Investigating bias (1)

Date: 

Sunday 16 September 2018 - 16:40 to 16:50

Location: 

Sidlaw Auditorium

All authors in correct order:

Page MJ1, Sterne JAC2, Higgins JPT2, on behalf of the development group for ROB-ME .1
1 School of Public Health and Preventive Medicine, Monash University, Australia
2 Population Health Sciences, Bristol Medical School, University of Bristol, UK
Presenting author and contact person

Presenting author:

Matthew Page

Contact person:

Matthew Page
Abstract text
Background: The credibility of evidence syntheses can be compromised by reporting biases. These include ‘publication bias’, when the probability that a study is published depends on its findings, and ‘outcome reporting bias’, when findings that are statistically non-significant are less likely to be reported. Reporting biases imply bias in evidence syntheses because missing results differ systematically from available results. Existing tools for assessing the risk of reporting biases in evidence syntheses are limited in terms of their scope, guidance for reaching risk of bias judgements and measurement properties.

Objectives: To develop a comprehensive tool for assessing risk of reporting biases in evidence syntheses.

Methods: A new tool, called ROB-ME (Risk Of Bias due to Missing Evidence), was developed in collaboration with an international group of 20 methodologists, statisticians, information specialists and review authors. Development of the tool was informed by a systematic review of the content and measurement properties of existing tools for assessing risk of reporting biases, and discussions at an in-person consensus meeting in April 2017 and a series of teleconferences.

Results: The ROB-ME tool includes multiple components. The early parts ask users to record, for each study identified, which outcomes were prespecified (e.g. in trial registries, protocols), and whether results are fully available, partially available, or not available for outcomes of interest. Users then specify a synthesis to be assessed for risk of bias. Signalling questions prompt users to consider several issues, such as how many studies are definitely missing from the synthesis because results are not available (despite the outcome having been measured), whether other studies are likely to be missing because of non-comprehensive searching, whether results are likely to be missing due to conflicts of interest of study investigators, and whether statistical/graphical methods suggest results are likely to be missing due to study findings. The tool includes algorithms to map responses to signalling questions to judgements about risk of bias.

Conclusions: We believe the new tool will offer considerable advantages over existing approaches for assessing risk of reporting biases in evidence syntheses.

Patient or healthcare consumer involvement: Not applicable.

Relevance to patients and consumers: 

Failing to consider reporting biases can result in the uptake of ineffective and harmful interventions. The recent Tamiflu saga illustrates this problem. Only after gaining access to withheld data from unpublished trials was a team of Cochrane reviewers able to determine that Tamiflu, a drug which the UK government spent £0.5bn on to prepare for flu pandemics, was ineffective. The new ROB-ME tool will enable those who perform evidence syntheses to provide more trustworthy conclusions, and thus has the potential to impact patients and the public through improved provision of care.