Implementation of ROBINS-I in non-randomized studies of interventions that have produced qualitatively conflicting results to randomized studies

ID: 

383

Session: 

Poster session 3

Date: 

Tuesday 18 September 2018 - 12:30 to 14:00

All authors in correct order:

McAleenan A1, Elbers R1, Lawlor D2, Martin R1, Reeves B3, Sterne J1, Higgins J1
1 Population Health Sciences, Bristol Medical School, University of Bristol, UK
2 MRC Integrative Epidemiology Unit, University of Bristol, UK
3 Clinical Trials and Evaluation Unit, School of Clinical Sciences, University of Bristol, UK
Presenting author and contact person

Presenting author:

Alexandra McAleenan

Contact person:

Alexandra McAleenan
Abstract text
Background:
Non-randomised studies of interventions (NRSI) are frequently included in Cochrane Reviews. The Risk Of Bias In Non- randomised Studies – of Interventions (ROBINS-I) tool was recently proposed to assess results from NRSI.

Objectives:
To determine whether the ROBINS-I tool can identify biases present in a selection of NRSI.

Methods:
We chose topics where the results of at least one NRSI conflicted with the results of large randomised controlled trials (RCTs) or systematic reviews of RCTs. We identified potential topics from 1) a systematic search for editorials containing terms suggestive of a discussion of conflicting results; 2) a review of meta-epidemiological studies comparing RCTs and NRSI; and 3) informal communications. Topics were eligible for inclusion if published RCTs and NRSI had addressed the same research question (assessed by an expert panel based on PICOs for each study), in which the RCT evidence was convincing, and which reached qualitatively different conclusions. We implemented ROBINS-I tool (BMJ 2016;355:i4919), in duplicate, to assess the NRSI.

Results:
We selected 14 topics. We found all NRSI assessed to be at serious or critical risk of bias in at least one domain. The most common reasons were confounding and bias due to selection of participants. There were also issues in other domains; for example, we found an example of outcomes being defined differently for intervention and comparator groups. Overall, the ROBINS-I tool picked up the likely reasons for discrepant results. However, we found that improvements could be made to the signalling questions relating to selection of participants into the study. Situations where immortal time was misclassified or excluded were not easy to detect, and a question regarding whether selection into each group was based on interventions given after the start of follow-up would be a useful addition.

Conclusions:
In these selected topics we found the NRSI to be at serious or critical risk of bias in at least one domain of ROBINS-I. Although we cannot be sure whether these biases were responsible for the differences seen, ROBINS-I appeared to have face validity in these cases. The tool would benefit from additional signalling questions to help assessors detect the presence of immortal time bias.

Patient or healthcare consumer involvement:
None.

Relevance to patients and consumers: 

Non-randomized studies of interventions (NRSIs) may be the only source of evidence regarding an intervention in situations where it is not practical/ethical to perform a randomized controlled trial (RCT), but also provide complementary evidence to RCTs regarding ‘real world’ settings, the long-term, and rare but serious side effects. To be able to make healthcare decisions on the basis of the results of NRSIs, we need to be assured that the results are not compromised by limitations or flaws in the study. ROBINS-I is a tool to assess risk of bias in NRSI. This work will help determine whether ROBINS-I can identify limitations in NRSI.