Misleading results even after adjustment for a dozen covariates: a comparison between 'real world' analyses and randomized controlled trials (RCTs) in telecardiology

Session: 

Oral session: Investigating bias (2)

Date: 

Monday 17 September 2018 - 14:40 to 14:50

Location: 

Carrick 2

All authors in correct order:

Schell LK1, Schulz A1, Angelescu K1, Glinz D2, Knelangen M1, Sauerland S1
1 Institute for Quality and Efficiency in Healthcare, Germany
2 Basel Institute for Clinical Epidemiology & Biostatistics, Switzerland
Presenting author and contact person

Presenting author:

Lisa Katharina Schell

Contact person:

Lisa Katharina Schell
Abstract text
Background: Advocates of 'real world evidence' argue that when adequately adjusting for bias, real world data might be as valid as those of randomized controlled trials (RCTs). Telecardiology allows a direct comparison, since both RCTs and large registries are available.

Objectives: In the field of telecardiology, we aimed to assess the validity of results derived from non-randomised studies (NRS). To this end, we comprehensively compared the mortality results of NRS with those of a meta-analysis including RCTs only.

Methods: In January 2018, we searched for NRS examining telemonitoring via implantable cardioverter-defibrillators (ICDs) or cardiac resynchronization therapies (CRTs) in people with heart failure. We considered the 10 largest NRS adjusting for confounders and published after 2007 (date of first RCT published) with a parallel control group and a minimal size of 100 participants per study arm. We compared the mortality results of the NRS with those of a systematic review with meta-analysis consisting of 17 RCTs including 10,130 participants (last search: August 2017).

Results: We identified six NRS (number of patients ranging from 312 to 185,778). Follow-up ranged from one to five years and is thus longer than those of the RCTs (maximum three years). All six NRS showed a significant beneficial association with all-cause-mortality for patients followed remotely (hazard ratios for death ranging from 0.19 to 0.67 (see Table 1)). Results were comparable for NRS adjusting for at least one disease variable and NRS without adjusting for disease variables. These significant results stand in sharp contrast to those of the meta-analysis of RCTs (odds ratio: 0.92 (95% confidence interval 0.81 to 1.05), P = 0.733).

Conclusions: The large discrepancy between the results of NRS and RCT suggests that residual confounding is a substantial problem of 'real world evidence'. In telecardiology, confounding by indication and self-selection are the most likely explanation. Therefore, even when adjusting for disease variables, bias in NRS is likely and their results should be interpreted with caution. Given their limited validity regarding efficacy, registries should be used for other purposes, especially when RCT evidence is already available.

Patient or healthcare consumer involvement: Patients were not involved in the presented analysis.

Attachments: 

PDF icon Table 1 – All-cause mortality of non-randomised studies in telecardiology.pdf

Relevance to patients and consumers: 

Our example illustrates that efficacy results of non-randomised studies should be interpreted with caution. Therefore, in order to make informed health decisions, patients and doctors should rely on results of randomized controlled studies to decide which interventions are beneficial and which ones are not.