A randomised trial of integrated machine-learning for systematic review 'Risk of bias' assessments

Session: 

Oral session: Innovative solutions to challenges of evidence production (2)

Date: 

Sunday 16 September 2018 - 17:00 to 17:20

Location: 

Fintry Auditorium

All authors in correct order:

Arno A1, Wallace B2, McKenzie J3, Marshall I4, Thomas J1, Elliott J3
1 Institute of Education, University College London, United Kingdom
2 College of Computer and Information Science, Northeastern University, United States
3 School of Public Health and Preventive Medicine, Monash University, Australia
4 Department of Primary Care and Public Health Sciences, King's College London, United Kingdom
Presenting author and contact person

Presenting author:

Anneliese Arno

Contact person:

Anneliese Arno
Abstract text
Background:
Assessment of study risk of bias (RoB) is a key step in a systematic review but is very time-consuming. RobotReviewer is an open-access platform which partially automates RoB assessment using machine-learning (ML) and natural language processing. Covidence is a cloud-based systematic review production tool, recommended by Cochrane for the production of Cochrane Reviews.

Objectives:
The purpose of this trial is to evaluate the use of ML in RoB assessments in systematic reviews. Specifically, we sought to:
1) evaluate the accuracy of RoB assessments produced using a combination of human effort and ML compared to those produced by humans alone; and
2) determine whether the person-time required to complete RoB using a combination of human effort and ML is less than those completed by humans alone.

Methods:
A two-arm, parallel-group randomised controlled trial is ongoing. Each included study has initial RoB assessment performed by two independent reviewers (Human 1 and Human 2). RobotReviewer assistance will be provided to one of the two reviewers, with this assistance assigned randomly (1:1) for each study. RoB consensus will be performed by a third reviewer (Human 3) blinded to the allocation of RobotReviewer assistance.

Results:
The trial commenced recruitment in January, 2018 and is ongoing. At current rate of recruitment, final results will be available by mid-2018 and the co-primary outcomes will be presented.

Conclusions:
Following completion of the trial, results will be disseminated through journal publications and conference presentations. Study data will be used to inform next steps in machine-learning design and promotion.

Patient and healthcare consumer involvement:
While patients and consumers are unlikely to interact directly with the results of this study, increased automation holds exciting potential for increased availability of high quality, up-to-date health information. This trial is being conducted under the larger goal of building living evidence processes to provide reliable, up to date evidence for health decision-making.

Relevance to patients and consumers: 

While patients and consumers are unlikely to interact directly with the results of this study, increased automation holds exciting potential for increased availability of high quality, up-to-date health information. This trial is being conducted under the larger goal of building living evidence processes to provide reliable, up to date evidence for health decision-making.