Comparison of AMSTAR 2 with ROBIS in systematic reviews including randomized and non-randomized studies

Session: 

Oral session: Inclusion of non-randomized designs

Date: 

Sunday 16 September 2018 - 15:00 to 15:10

Location: 

Sidlaw Auditorium

All authors in correct order:

Pieper D1, Puljak L2, Gonzalez Lorenzo M3, Minozzi S4
1 Witten/Herdecke University, Germany
2 University of Split School of Medicine, Croatia
3 University of Milan, Italy
4 Cochrane Review Group on Drugs and Alcohol, Department of Epidemiology, Lazio Regional Health Service, Italy
Presenting author and contact person

Presenting author:

Dawid Pieper

Contact person:

Dawid Pieper
Abstract text
Background: While the risk of bias in systematic reviews (ROBIS)-tool can be applied to all type of systematic reviews (SR), AMSTAR 2 enables more detailed assessment of systematic reviews that include randomised or non-randomised studies (NRS) of healthcare interventions, or both. Prior research indicates that including NRS into systematic reviews of therapeutic interventions is challenging.
Objectives: To report on a first experience with AMSTAR 2 and compare it with ROBIS when assessing SRs that include both RCTs and NRS while assessing validity, reliability and applicability.
Methods: Four raters assessed 30 randomly selected SRs taken from two samples of former projects. One sample consisted of Cochrane Reviews only, while the other only included non-Cochrane reviews. All SRs were assessed in the same order using AMSTAR 2 first, followed by ROBIS. For each question, domain and overall risk of bias, we calculated the Fleiss’ k for multiple IRR. We recorded the time to complete each tool as mean time spent by each reviewer on each review. We classified agreement as: poor (≤ 0.00), slight (0.01 to 0.20), fair (0.21 to 0.40), moderate (0.41 to 0.60), substantial (0.61 to 0.80), almost perfect (0.81 to 1.00).
Results: All raters assessed 12 SRs. IRR for ROBIS domains ranged from 0.09 to 0.38. IRR for overall risk of bias was fair (0.24, 95% CI 0.16 to 0.60). Median IRR for AMSTAR 2 was 0.49. Slight or poor agreement was obtained for item 4 (search strategy), 8 (adequately detailed description of included studies), 14 (explanation and discussion of heterogeneity) and 16 (conflict of interest at review level). The mean time to complete scoring was similar (AMSTAR 2: 19 minutes versus ROBIS: 17 minutes). However, huge differences were observed across raters. Results for all 30 SRs and for validity will be presented at the Colloquium.
Conclusions: On average IRR was much higher for AMSTAR 2 when compared to ROBIS. Taking into account that ROBIS has always been applied after AMSTAR 2, we assume that scoring for ROBIS takes more time in general. All raters experienced AMSTAR 2 to be satisfactorily applicable to SRs including RCTs and NRS. Some signalling questions in ROBIS were judged to be very difficult to assess.
Patient or healthcare consumer involvement: Due to the methodological character, patient or healthcare consumer involvement was not planned.

Attachments: 

PDF icon Abstract CC 2018_table.pdf

Relevance to patients and consumers: 

During that stage of our work, patient or healthcare consumer involvement was not planned. However, systematic reviews are considered as the best source of evidence for healthcare decisions. To ensure the best evidence, it is of great importance that decisions are made on the ground of trustworthy systematic reviews. This makes their assessment to be crucial for consumers and decision makers.