Causal modelling in RCTs: how results compare to ITT and other analytical methods – a meta-epidemiological analysis

Session: 

Oral session: Understanding and using evidence (2)

Date: 

Monday 17 September 2018 - 15:10 to 15:20

Location: 

Carrick 3

All authors in correct order:

Ewald H1, Speich B2, Ladanie A3, Bucher HC2, Ioannidis JPA4, Hemkens LG2
1 Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel; Swiss Tropical and Public Health Institute; University Medical Library, University of Basel, Switzerland
2 Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Switzerland
3 Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Swiss Tropical and Public Health Institute, Basel, Switzerland
4 Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Meta-Research Innovation Center at Stanford, Stanford School of Medicine, Department of Health Research and Policy, Stanford, Dept. of Biomedical Data, USA
Presenting author and contact person

Presenting author:

Hannah Ewald

Contact person:

Hannah Ewald
Abstract text
Background:
Analyses with marginal structural models (MSM) are increasingly used to estimate treatment effects. While more commonly used in randomised studies, they are increasingly applied in randomised clinical trials (RCTs).

Objectives:
To determine how MSM are used for healthcare decision-making in RCTs and to compare their treatment effect estimates with other reported estimates, such as those from intention-to-treat (ITT) analyses.

Methods:
We systematically searched PubMed, Scopus, citations of key references and ClinicalTrials.gov up to May 2017. Any RCTs reporting the effects of any healthcare intervention based on MSM and on at least one of ITT, as-treated or per protocol analysis were eligible. We systematically compared the MSM-based results with the effects from ITT and any other reported analyses (i.e. how frequently treatment effects from MSM and other analyses were in the same or in opposite directions, how often there was no overlap between the 95% confidence intervals (CIs) of the results, and how often the MSM-based effect lay within the 95% CI of the other effects). We determined the overall spread between the largest and smallest effect size derived from different analytical methods within one comparison. Focusing specifically on MSM-based versus ITT-based results, we determined which method reported on average more extreme results and determined the median magnitude of the effect sizes.

Results:
We included 12 RCTs published between 2002 and 2016, including a median of 1972 patients and reporting 138 analyses for 24 clinical questions. The analyses are ongoing and will be available at the time of the conference.

Conclusions:
Trials are increasingly analysed with more complex modelling, aiming to provide further useful insights on treatment effects beyond the ITT effect. It will be relevant for clinical decision-making to know whether these concepts come to different conclusions and how similar they are in general. This project provides first empirical evidence on this topic.

Patient or healthcare consumer involvement:
No patients or healthcare consumers were involved in this study.

Relevance to patients and consumers: 

Conventional analyses based on the intention-to-treat principle provide the most reliable estimate of the benefits and harms of initiating a treatment. However, the effect of a treatment under optimal adherence to the protocol (per protocol effect) may also be important to inform treatment choices. While unadjusted per protocol affects are subject to confounding bias, marginal structural modelling allows to adjust for post-randomization confounding but requires strong assumptions, more detailed data, and is more complex to plan, conduct, and report. A comparison of different analytical results may be of interest.