Taking evidence synthesis beyond its traditional role: a ‘rigorous speculation’ framework for predicting success of innovative treatments with early evidence bases




Poster session 1


Sunday 16 September 2018 - 12:30 to 14:00

All authors in correct order:

Helfand M1, Peterson K1, Anderson J1
1 VA Evidence-Based Synthesis Program Coordinating Center, USA
Presenting author and contact person

Presenting author:

Kim Peterson

Contact person:

Abstract text
Background: The US Veterans Health Administration (VHA) promotes innovation to provide veterans with the best healthcare. Thus, policy-makers are increasingly looking to evidence synthesis programs for decision support on innovative treatments with early evidence bases. Decision-makers make the point that while they need independent, unbiased information, traditional evidence syntheses take too long and address whether evidence is definitive, rather than whether an innovation is promising. In response, systematic reviews need to assess whether an early evidence base is exceptional in ways that may predict success, a concept we called ‘rigorous speculation’.
Objective: To conceptualize a framework for ‘rigorous speculation’ in evidence synthesis, an approach to improving health system decision support about innovative treatments with early evidence-bases.
Methods: In a June 2017 VA Evidence-based Synthesis Program (ESP) strategic planning meeting, discussions with health system decision makers led to identification of the ‘rigorous speculation’ concept. Based on the ESP Coordinating Center’s (ESP CC) experience piloting ‘rigorous speculation’ in rapid evidence reviews, we provide operational guidance on how to apply this approach.
Results: We identified the following processes for improving evaluation of early evidence-bases: 1) use of a new language that moves away from boilerplate language ('low strength evidence', etc); 2) improved description of case series’ ideal characteristics (e.g. unbiased patient selection, stability of treatment and patient characteristics pre-intervention, well-defined intervention, objective or blinded assessment); 3) identification of relevant historical evidence patterns for use in predicting success; 4) increasing linkages to relevant existing dataset resources; and 5) increased consideration of local context (e.g. access, feasibility, cost, patient preferences, etc.).
Conclusion: We propose this framework to increase consistency and transparency of evidence reviewers’ use of ‘rigorous speculation’, thus expanding their role as knowledge brokers for health system decision making. We will refine this framework based on further expert consultation and literature review.
Patient or healthcare consumer involvement: This framework considers patient/healthcare consumer preferences as a key local contextual factor.

Relevance to patients and consumers: 

VHA is increasingly promoting innovative treatments with early evidence-bases as a way to increase responsiveness to Veteran constituents' requests to expand treatment options to those that are important to them. This research discusses a framework for assessing whether early evidence bases for innovative treatments are exceptional in ways that may predict success-- a concept we called ‘rigorous speculation' -- thus expanding reviewers' role as knowledge brokers for health system decision-making and potentially helping patients receive the innovative treatments they've requested. Patients did not help us reach this statement.