Exploring discrepancies in the findings of 20 systematic reviews addressing the same PICO question: a case study

ID: 

349

Session: 

Poster session 3

Date: 

Tuesday 18 September 2018 - 12:30 to 14:00

All authors in correct order:

Khamis AM1, El Moheb M2, Nicolas J2, Refaat M3, Akl EA3
1 Clinical Research Institute, American University of Beirut, Lebanon
2 Faculty of Medicine, American University of Beirut, Lebanon
3 Department of Internal Medicine, American University of Beirut, Lebanon
Presenting author and contact person

Presenting author:

Assem Khamis

Contact person:

Assem Khamis
Abstract text
Background:
There are sometimes variation(s) in the meta-analytical findings between systematic reviews that address the same question and include the same studies.

Objective:
To assess and investigate the reasons for discrepancies between the results of meta-analyses from systematic reviews addressing the same question.

Methods:
We conducted a Cochrane Review on 'Implantable cardiac defibrillators for patients with non-ischemic cardiomyopathy' with all-cause mortality as the primary outcome. In parallel, we searched for non-Cochrane systematic reviews addressing the same question. We abstracted detailed information about number of studies, data extracted, analytical methods and effect estimates. We assessed the variation between the pooled estimates in meta-analyses through standard deviation calculation and we investigated the reasons for the discrepancies.

Results:
We identified 20 non-Cochrane systematic reviews published in 2017 that addressed the same question. All 20 reviews were published in 2017 and ran their searches after the publication of all included trials. The standard deviation for the variation of the pooled effect estimate for all cause-mortality across the 20 non-Cochrane systematic reviews was 4%.

We identified the following as reasons for the discrepancies in the findings of the 20 systematic reviews: 1) different included trials, 2) different effect estimates used (i.e. odds ratio (OR), relative risk (RR) or hazard ratio (HR)) in meta-analysis, 3) different effect estimates used from trials, 4) different analysis models used (random- or fixed-effect), 5) errors in data abstraction or entry, 6) errors in reporting of results, 7) obtaining additional data directly from trial authors and 8) use of different outcome timeframes. For example, only seven systematic reviews (35%) included all six trials in their meta-analyses. Half of the systematic reviews pooled RRs, while 40% and 15% pooled HRs and ORs, respectively.

Conclusion:
In a sample of 21 systematic reviews addressing the same question, there were frequent discrepancies between their meta-analytical results for the same outcome. It is debatable whether these discrepancies have implications for the interpretation of the findings. Reasons for the discrepancies were related either to errors or to choices in the analytical approach.

Patient or healthcare consumer involvement:
No involvement.

Relevance to patients and consumers: 

Systematic reviews have become the cornerstone of evidence-based decision making. However, using the inappropriate statistical methods could bias the results of those systematic reviews. At the same time, most of systematic reviews authors (consumers such as physicians, patients and policymakers have limited knowledge to evaluate the statistics methodology used in systematic reviews.