A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

Session: 

Oral session: Statistical methods (2)

Date: 

Sunday 16 September 2018 - 14:50 to 15:10

Location: 

Ochil 1

All authors in correct order:

Debray T1, Damen J1, Riley R2, Snell K2, Reitsma J1, Hooft L1, Collins G3, Moons K1
1 Julius Center for Health Sciences and Primary Care, Netherlands
2 Keele University, United Kingdom
3 Oxford University, United Kingdom
Presenting author and contact person

Presenting author:

Thomas Debray

Contact person:

Thomas Debray
Johanna Damen
Richard Riley
Abstract text
Background: It is widely recommended that any developed prediction model - diagnostic or prognostic - is validated externally in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment.

Objectives: To discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome.

Methods: We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c -statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: a meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed open source R package 'metamisc'.

Results: Frequentist and Bayesian meta-analysis methods often yielded similar summary estimates of prediction model performance. However, estimates of between-study heterogeneity and derived prediction intervals appeared more adequate when we applied Bayesian estimation methods.

Conclusions: Our empirical examples demonstrate that meta-analysis of prediction models is a feasible strategy despite the complex nature of corresponding studies. As developed prediction models are being validated increasingly often, and as the reporting quality is steadily improving, we anticipate that evidence synthesis of prediction model studies will become more commonplace in the near future. The R package metamisc is designed to facilitate this endeavor, and will be updated as new methods become available.

Patient or healthcare consumer involvement: The identification of relevant statistical methods was informed by previous experiences with systematic reviews of prognosis studies.

Relevance to patients and consumers: 

We propose new statistical methods to summarize existing evidence on prediction models. These models are commonly used to assess outcome risk (e.g. mortality) in individual patients, and to guide treatment strategies. The proposed methodology can be used to assess whether and to what extent prediction models are sufficiently accurate across different populations. This may help to improve medical decision making as it allows to identify potential flaws of existing and commonly used prediction models.