Creating living systematic reviews with citizen scientists and machine learning

Session: 

Oral session: Innovative solutions to challenges of evidence production (1)

Date: 

Sunday 16 September 2018 - 14:40 to 15:00

Location: 

Fintry Auditorium

All authors in correct order:

Elliott J1, Gordon C2, Noel-Storr A3, Thomas J4, Cohen A2, Hodder R5, Gilbert J2, Murano M1, Weiss K6, Synnot A1, Turner T1, Millard T1, Martin N7, Bridges C7, Casas J7, MacLehose H6, Chou R2, Wolfenden L5, Helfand M2
1 Cochrane Australia, Australia
2 Oregon Health Sciences University, USA
3 Cochrane Dementia and Cognitive Improvement, UK
4 University College London, UK
5 University of Newcastle, Australia
6 Cochrane, UK
7 Cochrane Heart, UK
Presenting author and contact person

Presenting author:

Julian Elliott

Contact person:

Julian Elliott
Abstract text
Background:
The barriers to patient and consumer participation in the production of systematic reviews are substantial. 'Citizen science' opens up new opportunities for co-creation of trustworthy, up to date evidence. In parallel, text mining and machine learning are now able to make meaningful contributions to systematic review production.

Objectives:
The Next Generation Evidence Project, funded by the Robert Wood Johnson Foundation, aimed to assess the feasibility of producing living systematic reviews with the participation of patients and consumers, and contribution from text mining and machine learning.

Methods:
New features were developed on Cochrane Crowd, Cochrane’s citizen science platform, to improve accessibility for a broad spectrum of individuals. Cochrane Crowd was promoted to patients and consumers in the USA in partnership with consumer and professional health organisations, and consumers from these organisations evaluated the Cochrane Crowd platform. Cochrane Crowd citizen scientists, together with machine learning systems, assisted systematic review teams to develop and maintain two living systematic reviews in the field of child health.

Results:
The Cochrane Crowd platform was developed and promoted widely to USA health consumer organisations. Specific features included:
- a Learning Zone where contributors can undertake brief training to increase understanding of key research concepts;
- topic filters so contributors can find content relevant to their specific interests, e.g. dementia, diabetes or cancer;
- micro-tasks more suitable for beginners, such as the identification and classification of data tables in full-text articles. Text mining and machine learning systems were developed to automatically extract structured data from these tables.

Two living systematic reviews were developed and updated over time with support from Cochrane Crowd citizen scientists and the machine learning systems. Crowd members screened 1600 abstracts in 4.5 hours and identified 3674 tables in 24 hours.

Conclusions:
Patients and consumers can play crucial roles in evidence synthesis, and enable rigorous systematic reviews to frequently and rapidly incorporate the latest evidence.

Patient or healthcare consumer involvement:
Patients and healthcare consumers contributed to this project and its evaluation.

Relevance to patients and consumers: 

Patients and consumers can face significant barriers to participation in the production of systematic reviews. ‘Citizen science’ offers an entry point with a low threshold for participation. In this project we assessed the feasibility of a citizen science approach based on Cochrane Crowd, and the use of text mining and machine learning, to develop and maintain trustworthy and frequently updated systematic reviews, an approach termed ‘living systematic review’.