PROBAST: a 'Risk of bias' tool for prediction modelling studies

Session: 

Oral session: Investigating bias (3)

Date: 

Tuesday 18 September 2018 - 12:00 to 12:20

Location: 

Carrick 1

All authors in correct order:

Wolff R1, Moons KGM2, Riley RD3, Whiting PF4, Westwood M1, Collins GS5, Reitsma JB2, Kleijnen J1, Mallett S6
1 Kleijnen Systematic Reviews Ltd, UK
2 University of Utrecht, The Netherlands
3 University of Keele, UK
4 University of Bristol, UK
5 University of Oxford, UK
6 University of Birmingham, UK
Presenting author and contact person

Presenting author:

Robert Wolff

Contact person:

Robert Wolff
Abstract text
Background: Quality assessment of included studies is a crucial step in any systematic review (SR). Review and synthesis of prediction modelling studies is evolving and a tool facilitating quality assessment for prognostic and diagnostic prediction modelling studies is needed.
Objectives: To introduce PROBAST, a tool for assessing the risk of bias and applicability of prediction modelling studies in a SR.
Methods: A Delphi process, involving 40 prediction research experts was used until agreement on the content of the final tool was reached. Existing initiatives such as the REMARK and TRIPOD reporting guidelines for prediction research formed part of the evidence base for the tool development. The scope of PROBAST was determined with consideration of existing tools such as QUIPS and QUADAS 2.
Results: After six rounds of the Delphi procedure, a final tool was developed which utilises a domain-based structure supported by signalling questions similar to QUADAS 2. PROBAST assesses the risk of bias and applicability of prediction modelling studies. Risk of bias refers to any shortcomings in the study design, conduct or analysis leading to systematically distorted estimates of predictive performance or an inadequate model to address the research question. The predictive performance is typically evaluated using calibration, discrimination and sometimes classification measures. Assessment of applicability examines whether the prediction model development or validation study matches the systematic review question in terms of the target population, predictors, or outcomes of interest
PROBAST comprises four domains (Participant selection; Predictors; Outcome; Analysis) and 20 signalling questions grouped within these domains.
Conclusions: PROBAST can be used to assess the quality of prediction modelling studies included in a SR. The presentation will give an overview of the development process and introduce the final tool.
Patient or healthcare consumer involvement: A wide range of stakeholders was involved in the development and testing of PROBAST resulting in a tool that is methodologically sound, user-friendly, and relevant in various contexts. Stakeholders are directly involved in producing guidelines and shared-decision making tools, i.e. we envisage that PROBAST will be used in relevant research projects supporting and involving healthcare consumers.

Relevance to patients and consumers: 

Prediction models in healthcare include models that aim to predict a particular outcome, e.g. a disease or disorder. These models can be used to test for current or future outcomes (e.g. https://qrisk.org/2017/). Therefore, prediction models can inform healthcare professionals as well as healthcare consumers whether further testing is required, about the best treatment, or preventive lifestyle changes. PROBAST is a tool to assess the quality (risk of bias) and relevance (applicability) of prediction models hence helping to identify the best tool for the job, i.e. the highest quality and most relevant prediction model for a certain question.