Statistical methods for reliably updating meta-analyses

Session: 

Oral session: Statistical methods (2)

Date: 

Sunday 16 September 2018 - 14:30 to 14:50

Location: 

Ochil 1

All authors in correct order:

Simmonds M1
1 Centre for Reviews and Dissemination, University of York, UK
Presenting author and contact person

Presenting author:

Mark Simmonds

Contact person:

Mark Simmonds
Abstract text
Background:
Cochrane Reviews require updating as new studies become available, requiring multiple, repeated meta-analyses to analyse these studies. This increases the risk of attaining spurious 'statistical significance' in each analysis. As updating of reviews becomes both more common and more frequent there is an increasing risk that meta-analysis results may be misinterpreted, particularly if readers are unaware of the updating process.

Objectives:
To discuss the findings of a Cochrane-funded project, which has formally assessed a number of statistical methods for appropriately updating meta-analyses.

Methods:
Methods assessed include standard random-effects analysis, trial sequential analysis and sequential meta-analysis. These have been applied to a range of recently updated Cochrane Reviews and tested in simulation studies.

Results:
The use of standard meta-analysis methods when updating Cochrane Reviews does increase the risk of spurious findings, which would be overturned at later updates. This risk is substantial when a meta-analysis includes few trials, or the trials are very heterogeneous in their findings. Trial sequential analysis and sequential meta-analysis, and potentially Bayesian analyses, are the only statistical methods found to reduce this risk. This benefit comes at the cost of requiring more trials or review updates to reach a firm conclusion.

Conclusions:
All Cochrane Reviews should consider the risk of drawing incorrect conclusions due to repeated updating and analysis over time, particularly where reviews are updated frequently. One simple option is to consider whether the total sample size of the review is adequate. Trial sequential analysis and sequential meta-analysis offer formal statistical techniques for reliable updating of reviews, but come with greater statistical complexity that may be unnecessary in reviews with infrequent updates. These methods may be of particular importance where reviews guide clinical decision-making, where having robust evidence that an intervention is useful is essential.

Patient or healthcare consumer involvement:
This project was funded by the Cochrane Methods Innovation Fund and had no direct patient involvement. However, patients rely on reviews being up-to-date, and that medical decisions based on reviews are justified given the evidence and will not change in the future.

Relevance to patients and consumers: 

Patients and physicians rely on Cochrane reviews being up-to-date to ensure best medical practice. However, frequent updating of reviews increases the risk that the results of statistical analyses will be misinterpreted. Cochrane reviews must use the best and most appropriate statistical methods to avoid any such misinterpretation. This presentation discusses the findings of a project to investigate what statistical methods sholud be used when updating reviews, which was identified as an important area in need of research, and was funded by, the Cochrane Methods Innovation Fund. No patient partners were involved in this research.