Background: The credibility of evidence syntheses can be compromised by reporting biases. These include ‘publication bias’, when the probability that a study is published depends on its findings, and ‘outcome reporting bias’, when findings that are statistically non-significant are less likely to be reported. Reporting biases imply bias in evidence syntheses because missing results differ systematically from available results. Existing tools for assessing the risk of reporting biases in evidence syntheses are limited in terms of their scope, guidance for reaching risk of bias judgements and measurement properties.
Objectives: To develop a comprehensive tool for assessing risk of reporting biases in evidence syntheses.
Methods: A new tool, called ROB-ME (Risk Of Bias due to Missing Evidence), was developed in collaboration with an international group of 20 methodologists, statisticians, information specialists and review authors. Development of the tool was informed by a systematic review of the content and measurement properties of existing tools for assessing risk of reporting biases, and discussions at an in-person consensus meeting in April 2017 and a series of teleconferences.
Results: The ROB-ME tool includes multiple components. The early parts ask users to record, for each study identified, which outcomes were prespecified (e.g. in trial registries, protocols), and whether results are fully available, partially available, or not available for outcomes of interest. Users then specify a synthesis to be assessed for risk of bias. Signalling questions prompt users to consider several issues, such as how many studies are definitely missing from the synthesis because results are not available (despite the outcome having been measured), whether other studies are likely to be missing because of non-comprehensive searching, whether results are likely to be missing due to conflicts of interest of study investigators, and whether statistical/graphical methods suggest results are likely to be missing due to study findings. The tool includes algorithms to map responses to signalling questions to judgements about risk of bias.
Conclusions: We believe the new tool will offer considerable advantages over existing approaches for assessing risk of reporting biases in evidence syntheses.
Patient or healthcare consumer involvement: Not applicable.