An empirically defined decision tree to predict systematic reviews at risk of change in conclusion

Session: 

Oral session: Searching and information retrieval (1)

Date: 

Sunday 16 September 2018 - 11:50 to 12:00

Location: 

All authors in correct order:

Bashir R1, Surian D1, Dunn AG1
1 Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Australia
Presenting author and contact person

Presenting author:

Rabia Bashir

Contact person:

Abstract text
Background:
Systematic reviews are resource-intensive so it is important to focus on reviewing interventions for which new evidence might warrant a change in practice.

Objectives:
To determine whether basic information about new relevant trials can be used to estimate the risk of a change in conclusion in published systematic reviews.

Methods:
We identified systematic reviews that had updates published between October 2016 and December 2017, including pairs with consistent search strategies, inclusion criteria and outcomes, and where most included studies were trials. We analysed reviews that added new trials and reported the numbers of participants. We extracted: the total number of trials and participants in the original review; the time between the two search dates; and the completeness - the number of participants in the original review as a proportion of the number of participants in the update. A change in conclusion was defined by a change in significance of a primary safety or efficacy outcome (evaluated independently by two investigators; disagreements resolved by discussion). We trained a Classification and Regression Tree to predict (five-fold cross validation) a change in conclusion using some or all of the factors, reporting average precision and recall.

Results:
We analysed 63 pairs of reviews, of which 20 reported a change in conclusion in the update. Using the number of trials/participants in the original review and time elapsed to the new search date, the decision tree produced an average precision of 40% and a recall of 70%. After adding completeness to the decision tree, this increased to an average precision of 60% and a recall of 90%. The decision tree showed that reviews were most at risk of a change in conclusion when completeness was low (≤ 13.5%), the original review had fewer trials (< 23) and more time had elapsed (> 53 months).

Conclusions:
An empirically defined decision tree using simple information extracted from a published systematic review and basic information about trials that may be relevant can estimate the risk of a change in conclusion. The results can be used to better target resources for updating systematic reviews and would benefit patients by identifying evidence reversals earlier.

Attachments: 

Relevance to patients and consumers: 

A relatively small proportion of systematic reviews produce a change in conclusion that would warrant a change in clinical practice. Without actually undertaking the systematic review it is challenging to guess whether such a change is likely. If we are better able to predict which systematic reviews are at risk of a change in conclusion we could target resources to those updates and reduce the time taken to identify clinical questions where changes in practice are warranted, and provide a better basis from which patients and clinicians can together make decisions about their health.