Managing a very large systematic review of observational studies and randomized trials without compromising on methodological rigor

Session: 

Oral session: Innovative solutions to challenges of evidence production (4)

Date: 

Tuesday 18 September 2018 - 15:00 to 15:10

Location: 

Carrick 1

All authors in correct order:

Zeraatkar D1, Rabassa M2, Valli C2, Vernooij RWM3, El Dib R4, Bala MM5, Alonso-Coello P6, Guyatt GH7, Johnston BC8
1 Department of Health Research Methods, Evidence and Impact, McMaster University, Canada
2 Iberoamerican Cochrane Centre Barcelona, Biomedical Research Institute San Pau (IIB Sant Pau), Spain
3 Department of Community Health and Epidemiology, Faculty of Medicine, Dalhousie University; Department of Research, Netherlands Comprehensive Cancer Organisation (IKNL), The Netherlands
4 Department of Community Health and Epidemiology, Faculty of Medicine, Dalhousie University; Institute of Science and Technology, Unesp - Univ Estadual Paulista, Brazil
5 Division of Epidemiology and Preventive Medicine, Department of Hygiene and Dietetics, Jagiellonian University Medical College, Poland
6 Department of Health Research Methods, Evidence and Impact, McMaster University; Iberoamerican Cochrane Centre Barcelona, Biomedical Research Institute San Pau (IIB Sant Pau); CIBER de Epidemiología y Salud Pública (CIBERESP), Spain
7 Department of Health Research Methods, Evidence and Impact, McMaster University; Department of Medicine, McMaster University, Canada
8 Department of Health Research Methods, Evidence and Impact, McMaster University; Department of Community Health and Epidemiology, Faculty of Medicine, Dalhousie University, Canada
Presenting author and contact person

Presenting author:

Dena Zeraatkar

Contact person:

Dena Zeraatkar
Abstract text
Background: Guideline development often requires the consideration of a large body of evidence, including many subgroups and outcomes. In such situations, systematic review authors may face trade-offs between rigor and feasibility. We confronted this situation in conducting a systematic review addressing the health effects of red meat consumption to inform a nutrition guideline.

Methods: Initially, we planned to include all randomized trials and observational studies reporting on the health effects of red meat. A preliminary search suggested that there would be more than a thousand articles meeting our eligibility criteria. We considered restricting eligibility on the basis of the number of included participants or study designs at lower risk of bias. Ultimately, we chose to include all randomized trials and cohort studies enrolling at least 1000 participants. We also refined our outcomes and subgroups of interest to make the review as manageable as possible, while still retaining all information that would be useful to the panel in making recommendations. As an example, with consensus of the guideline panel, we chose to only extract data on cancer outcomes with hypothesized associations with red meat consumption, rather than all cancer outcomes that we had initially intended to present. We also chose to restrict abstraction of data on surrogate outcomes to randomized trials.

Results: Of 10,406 references retrieved for title and abstract screening, we selected 1393 for full-text screening, of which 443 proved eligible. Data abstraction for these references is in progress and with the modifications we have instituted, has proved manageable. We will present our experiences in managing this large systematic review, including problems we have encountered and our approaches in dealing with them with the objective of maintaining high methodological rigor despite the size of the review.

Discussion: With exponential growth in the number of publications, more researchers are likely to find themselves conducting very large systematic reviews. While a plethora of methodological guidance exists for systematic reviewers, our work augments what to now has been limited guidance on dealing with the logistical challenges of very large reviews.

Patient/healthcare consumer involvement: Public representatives were involved in developing the research questions.

Relevance to patients and consumers: 

Public representatives were involved in the development of the research questions and subgroups of interest of our systematic review. We plan to highlight the role public representatives played in refining our research questions and subgroups of interest to make the systematic review manageable while still retaining all information that will be useful to them as potential consumers.