A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR

Session: 

Oral session: Overviews of reviews and network meta-analysis (2)

Date: 

Monday 17 September 2018 - 14:00 to 14:10

Location: 

Fintry Auditorium

All authors in correct order:

Perry R1, Leach V2, Davies P2, Penfold C1, Ness A1
1 NIHR Bristol Biomedical Research Centre – Nutrition Theme, UK
2 The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, UK
Presenting author and contact person

Presenting author:

Philippa Davies

Contact person:

Philippa Davies
Abstract text
Background:
AMSTAR (A MeaSurement Tool to Assess systematic Reviews) is an 11-item tool that has been used frequently to check the quality of a systematic review and determine whether the most important elements are reported. The recently developed ROBIS (Risk of Bias in Systematic Reviews) tool was designed to evaluate the level of bias present within a systematic review.

Objectives:
The aim of this work was to compare AMSTART and ROBIS as part of an overview of systematic reviews of various complementary and alternative medicine (CAM) for fibromyalgia with regard to content, inter-rater reliability and usability.

Methods:
We searched five databases - MEDLINE, Embase, AMED (via OVID), Web of Science and CENTRAL - from their inception to December 2015. Two reviewers independently assessed each review using both tools. The inter-rater reliability of all sub-sections and the overall ratings using each instrument (AMSTAR and ROBIS) were calculated using the unweighted kappa statistic and percentage agreement.

Results:
We included 15 systematic reviews in this overview. In general, there was a high level of consistency between the overall AMSTAR scale and the ROBIS risk of bias rating. There were discrepancies with three reviews. Overall, five reviews scored 6 or above on the AMSTAR scale. The inter-rater agreement was good (Ƙ = 0.70, 95% confidence interval (CI) 0.60 to 0.85), with 83.6% agreement between two raters. Using the ROBIS tool, seven reviews achieved a low risk of bias rating, six a high rating and two were rated as having an unclear risk of bias. The inter-rater agreement was fair (Ƙ = 0.32, 95% CI 0.13 to 0.48), with 60.0% agreement between the two raters.

Conclusions:
ROBIS is an effective tool for assessing risk of bias in systematic reviews, but the agreement was less good and it is more difficult to use compared to AMSTAR. Reviews that included a meta-analysis were easier to rate; however, further developmental work could improve its use in reviews without a formal synthesis.

Patient or healthcare consumer involvement:
No patients or healthcare consumers were involved in this piece of methodological work.

Relevance to patients and consumers: 

This was a methodological piece of work conducted by systematic reviewers. No patients or consumers were involved. Systematic reviews differ in quality and those that are not well conducted may be misleading or even harmful for patients. Whilst the work may not be of direct interest, he evaluation of tools for appraising the risk of bias in reviews are therefore of relevance patients and consumers in that they have the potential to improve the care that patients receive.