Development of a new instrument to assess the credibility of effect modification

Session: 

Oral session: Innovative solutions to challenges of evidence production (4)

Date: 

Tuesday 18 September 2018 - 14:00 to 14:20

Location: 

Carrick 1

All authors in correct order:

Schandelmaier S1, Briel M2, Sun X3, Dahabreh I4, Walsh M1, Thabane L1, Hoes AW5, Gagnier JJ6, Sauerbrei W7, Hayward RA6, Ioannidis JP8, Van der Heijden GJ9, Borenstein M10, Schmid CH4, Varadhan R11, Guyatt GH1
1 McMaster University, Canada
2 University Hospital Basel, Switzerland
3 Cochrane China, China
4 Brown University, USA
5 University Medical Center Utrecht, The Netherlands
6 University of Michigan, USA
7 Freiburg University, Germany
8 Stanford University, USA
9 University of Amsterdam, The Netherlands
10 Biostat, USA
11 John Hopkins University, USA
Presenting author and contact person

Presenting author:

Stefan Schandelmaier

Contact person:

Stefan Schandelmaier
Abstract text
Background:
Analyses of effect modification in randomized trials and meta-analyses are important to inform personalized medicine, but can result in spurious inferences. For example, debates regarding the credibility of subgroup analyses and meta-regression are often contentious. A formal, consensus-based, and user-tested instrument to assess the credibility of effect modification in randomized trials and meta-analyses remains unavailable. We are currently developing such an instrument.

Methods:
We are following a rigorous instrument development process. Our systematic survey of the methodological literature on the credibility of effect modification analyses identified 36 candidate items for the new instrument (Table 1). In collaboration with 10 experts (all authors of at least two methodological publications addressing effect modification), we are currently conducting a consensus study in which the team provides feedback on sequential versions of a new instrument. In addition, we will involve potential users of the new instrument including trialists, systematic reviewers, guideline developers, and journals editors. They will apply the draft instrument to a sample of subgroup analyses using formal user testing methods. Once the instrument is finalized, we will perform a sensibility testing with a second group of experts, and a reliability study with a second group of users.

Results:
Having presented the concept and candidate items at the last Cochrane Colloquium, we will now present the instrument that emerges from the consensus study and the user testing. We will demonstrate the use of the credibility instrument in a number of examples and discuss its compatibility with Cochrane Reviews and the GRADE approach.

Conclusion:
We expect that the new instrument will improve the analysis, interpretation, and reporting of effect modification in individual trials and meta-analyses.

Patient or healthcare consumer involvement: Not applicable

Attachments: 

PDF icon Table 1.pdf

Relevance to patients and consumers: 

The new instrument will help improve the quality of personalized health care which depends on the credibility of effect modification.