Development and application of the RD-10 critical appraisal checklist in a systematic review of 181 regression discontinuity studies

Session: 

Oral session: Inclusion of non-randomized designs

Date: 

Sunday 16 September 2018 - 15:10 to 15:20

Location: 

Sidlaw Auditorium

All authors in correct order:

Hilton Boon M1, Thomson H1, Campbell M1, Craig P1, Moore L1
1 MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, UK
Presenting author and contact person

Presenting author:

Michele Hilton Boon

Contact person:

Michele Hilton Boon
Abstract text
Background: Natural experiments are of increasing interest because of their potential to address confounding and selection, leading to stronger evidence of causal effects. Regression discontinuity (RD) designs can be used to evaluate an intervention or exposure in which a cut-off rule determines treatment allocation. In order to make the best use of these studies in systematic reviews, critical appraisal tools need to reflect the strengths and limitations specific to the design.

Objectives: We aimed to:
1) test the US Department of Education What Works Clearinghouse Standards for RD (WWC) using a sample of public health studies;
2) develop a critical appraisal checklist for RD that is applicable to health research; and
3) apply the checklist in a comprehensive review of RD studies.

Methods: We conducted a systematic review of RD studies of health outcomes following a protocol registered with PROSPERO (CRD42015025117). We searched textbooks, methodological papers and the Internet for potential quality assessment tools. We piloted the only published RD-specific assessment tool (WWC) and evaluated 17 studies of minimum legal drinking age (MLDA) legislation. We developed a 10-item checklist for RD appraisal (RD-10) and assessed the quality of 181 studies included in the systematic review.

Results: Of the 17 MLDA studies, 16 failed to meet WWC standard 2 (reporting of attrition) and therefore failed the overall quality standard. WWC produced limited description of study quality, was of little use in distinguishing between higher and lower quality studies, and made no allowances for cross-sectional or retrospective applications of the design. We adapted the WWC standards to produce 10 individual criteria that could be judged as yes, no, or unclear. The majority of studies (160/181; 88.4%) provided a narrative explaining how the cut-off rule was implemented. Just over half of the studies (93/181; 51.4%) specified whether the cut-off value was used only to assign participants to the treatment of interest. Fewer than half the studies reported a density test or a falsification test (74/181; 40.9%). Only 5% (9/181) of studies fully met the 10 criteria.

Conclusions: Compared to WWC, RD-10 is easier to use, produces a more detailed description of quality, and is more applicable to the retrospective RD designs commonly seen in health research.

Attachments: 

Image icon RD10_results.png

Relevance to patients and consumers: 

There is growing interest in natural experiments, non-randomised study designs that allow the effects of interventions and policies to be evaluated in a robust way using real-world data. Among these designs, regression discontinuity (RD) is increasingly used in medicine and health policy to evaluate interventions that depend on applying cut-off rules (such as age, blood pressure or income). But how do we know if such a design has been correctly implemented and well reported? Patients and consumers who are involved in systematic reviews or policy evaluation will be interested to learn more about this design and how its quality can be assessed.