Non-randomised studies of interventions (NRSI) are frequently included in Cochrane Reviews. The Risk Of Bias In Non- randomised Studies – of Interventions (ROBINS-I) tool was recently proposed to assess results from NRSI.
To determine whether the ROBINS-I tool can identify biases present in a selection of NRSI.
We chose topics where the results of at least one NRSI conflicted with the results of large randomised controlled trials (RCTs) or systematic reviews of RCTs. We identified potential topics from 1) a systematic search for editorials containing terms suggestive of a discussion of conflicting results; 2) a review of meta-epidemiological studies comparing RCTs and NRSI; and 3) informal communications. Topics were eligible for inclusion if published RCTs and NRSI had addressed the same research question (assessed by an expert panel based on PICOs for each study), in which the RCT evidence was convincing, and which reached qualitatively different conclusions. We implemented ROBINS-I tool (BMJ 2016;355:i4919), in duplicate, to assess the NRSI.
We selected 14 topics. We found all NRSI assessed to be at serious or critical risk of bias in at least one domain. The most common reasons were confounding and bias due to selection of participants. There were also issues in other domains; for example, we found an example of outcomes being defined differently for intervention and comparator groups. Overall, the ROBINS-I tool picked up the likely reasons for discrepant results. However, we found that improvements could be made to the signalling questions relating to selection of participants into the study. Situations where immortal time was misclassified or excluded were not easy to detect, and a question regarding whether selection into each group was based on interventions given after the start of follow-up would be a useful addition.
In these selected topics we found the NRSI to be at serious or critical risk of bias in at least one domain of ROBINS-I. Although we cannot be sure whether these biases were responsible for the differences seen, ROBINS-I appeared to have face validity in these cases. The tool would benefit from additional signalling questions to help assessors detect the presence of immortal time bias.
Patient or healthcare consumer involvement: