Training your robot: best practices for leveraging artificial intelligence (AI) in reference screening

Workshop category: 

  • Innovative solutions to challenges of evidence production
Date and Location

Date: 

Monday 17 September 2018 - 14:00 to 15:30

Location: 

Tinto
Contact persons and facilitators

Contact person:

Chantelle Garritty
Ian Stefanison

Facilitators:

Garritty C1, Stefanison I2, O'Connor A3
1 Cochrane Rapid Reviews Methods Group (Ottawa Hospital Research Institute), Canada
2 Evidence Partners, Canada
3 Iowa State University, USA
Target audience

Target audience: 

broad audience of participants who are interested in automation of systematic review processes

Level of difficulty: 

Basic
Type of workshop

Type of workshop : 

Training
Abstract

Abstract:

Background:
We are on the cusp of an important change in the way systematic reviews are produced. Artificial Intelligence (AI) and Natural Language Processing (NLP) will soon be assisting with our screening and data extraction processes. The open question remains how and where these technologies will be applied in order to yield the most value to those that produce and use systematic reviews (SRs) without jeopardizing quality, transparency and reproducibility.

Objectives:
The learning objectives of this workshop are to:
- develop a working knowledge of AI and NLP in the context of reference screening and data extraction;
- understand the current thinking on where AI fits in the systematic review process;
- understand the current limitations of AI and NLP;
- develop knowledge of the various NLP-based tools available to assist with reference screening and data extraction;
- provide an opportunity to practice using AI in the review process.

Description:
The workshop will start with a brief didactic session designed to give participants a solid baseline for the theory behind AI, NLP and classifiers in the SR context. The session will provide an overview of available research on the subject, focusing on the practical implications of the technology. The didactic session will conclude with a review of available technologies and tools that reviewers can experiment with as they explore the potential of AI for use in their own reviews.
The interactive portion of the workshop will allow participants to experiment with different applications of AI technology in the review process. The DistillerSR platform will be used as the training tool for this session and attendees will be provided with accounts and sample projects on which to work.
Attendees will do the following during the ‘hands-on’ portion of the workshop:
- experiment with training and AI to perform screening;
- use an AI to rank references in order of likeliness for inclusion;
- use an AI to assist with screening;
- use an AI review screened references for false exclusions.
Importantly, the session will examine best practices for training NLP classifiers and for developing AI-friendly protocols. Various methods for limiting the impact of potential AI errors will also be discussed.

Relevance to patients and consumers: 

Patients and healthcare consumers are welcome to attend to learn more about the growing use of artificial intelligence (AI) in the production of medical evidence. The inclusion of patients and consumers will provide an opportunity to discuss concerns around autonomous decision making and the potential for introduction of bias as a result of AI use. This would also facilitate a more general discussion about the limitations of AI and possible policies or guidelines that should govern its use in our field.