Agreements about risk of bias in randomized controlled trials on different interdental cleaning aids

ID: 

169

Session: 

Poster session 1

Date: 

Sunday 16 September 2018 - 12:30 to 14:00

All authors in correct order:

Poklepovic Pericic T1, Sambunjak D2, Johnson T3
1 Cochrane Croatia, University of Split School of Medicine, Croatia
2 Department of Nursing, Croatian Catholic University, Zagreb, Croatia
3 Yorkshire Area, Faculty of General Dental Practice, York, UK
Presenting author and contact person

Presenting author:

Trevor Johnson

Contact person:

Tina Poklepovic Pericic
Trevor Johnson
Abstract text
Background: Evaluation of internal validity of randomized controlled trials (RCTs) included in Cochrane systematic reviews (CSR) implies critical assessment of the risk of bias (RoB) in the studies' results, and is regarded an essential component of a review process. The Cochrane RoB tool comprises judgments of the level of RoB as low, unclear or high for seven different domains, with accompanying explanations for each judgment. It is recommended that it be used by two authors independently, a content expert and a methodologist.

Objectives: The aim of this study was to investigate the inter-rater reliability in assessing RoB in RCTs included in CSRs among review authors with different methodological and clinical backgrounds.

Methods: Three CSRs on interdental cleaning, one of which is an ongoing review, have been conduced with 44 included RCTs. Three review authors enrolled in all three CSRs, two content experts (TMJ, TPP), and a methodologist (DS), performed independent RoB assessments using the Cochrane RoB Tool for 31 out of 44 RCTs. Level of agreement in RoB assessments between review authors was analysed using the Advanced Analytics LLC software for Excel Windows (AGREESTAT 2011, Advanced Analytics, Maryland, USA) for calculating the Cohen kappa statistics.

Results: Preliminary results showed moderate agreement between a methodologist and both content experts, with = 0.50329, standard error (SE) = 0.05224 (95% CI 0.4 to 0.606) and the percent of agreement of 0.71 (95% CI 0.651 to 0.775) between DS and TMJ, and = 0.51280, SE = 0.050568 (95% CI 0.413 to 0.612), the percent of agreement of 0.71770, SE = 0.03121 (95% CI 0.656 to 0.779) between DS and TPP. Substantial agreement was found between the two content experts, with = 0.78248, SE = 0.039729 (95% CI 0.704 to 0.861) and the percentage of agreement of 0.87560, SE = 0.022884 (95% CI 0.83 to 0.921).

Conclusions: Content experts have different opinions about RoB compared to a methodology expert. Different opinions are beneficial as they provide a different perspective on the potential source and impact of bias influencing the results of RCTs and subsequently the findings of a CSR.

Patient or healthcare consumer involvement: This research has been performed by Cochrane members. Dental, especially oral hygiene topics are specific because we can all relate to them as patients or consumers.

Relevance to patients and consumers: 

When independently assessing risk of bias in studies included in Cochrane systematic reviews, methodologists and content experts reach different conclusions about bias. Having authors with different backgrounds involved in risk of bias assessment is important as it allows a different perspective on the level, the impact and the scale of bias, that can impact quality of evidence and conclusions of a review on which everyday health decisions are being made. Differences in bias assessments give space to open discussions about why they happen, how disagreements are to be resolved, and on it's clinical implications.