Background: Meta-analysis (MA) is widely used in research synthesis, however, the decision to perform a quantitative synthesis is not always easy. The clinical and methodological diversity across studies must be considered in the decision to combine data. There is discordance regarding the combination of observational data with randomized controlled trials (RCTs) data. Some argue that low power from lack of RCTs should be compensated for by inclusion of observational studies.

Objectives: To compare the criteria for inclusion of studies and the statistical analysis decision between Cochrane systematic reviews (SRs) and SRs published in other high impact factor (IF) journals.

Methods: We performed a cross-sectional study that included all SRs of interventions published in high IF journals in 2015 (IF - JCR > 11,862, 2015) and a paired random sample from 2015 published Cochrane SRs. Two authors extracted data about the SRs and the statistical analysis performed by them.

Results: We included 134 SRs. Regarding the number of included studies in a MA, we found a higher number (P < 0.005) in non-Cochrane SRs (mean 16.61; SD 14.690) when compared to Cochrane SRs (mean 7.62; SD 11.54). There was a higher number of participants (P < 0.005) in non-Cochrane SRS (mean 2009.71; SD 4335.29 versus 56601.93; 135452.14 in Cochrane SRs). The predominant type of study design included in MA was the RCT in both groups, however, there was a higher number of inclusion of at least one observational study in non-Cochrane SRs (P < 0.005).

Conclusions: Our analysis is mainly descriptive, but it raises some concerns. The number of MA and studies included was higher in non-Cochrane SRs. The Cochrane methodology recommends cautious when including other types of studies rather than RCTs in intervention reviews. There is a lot of discussion regarding the inclusion of other types of design, but it is a fact that meta-analytic approaches including different types of study designs are associated with an increase in the risk of bias. Further studies are needed to evaluate the interpretation and generalization of the results from SRs that include observational studies to evaluate the safety and efficacy of interventions in health care.

Patient or healthcare consumer involvement: The performance and interpretation of quantitative analysis are imperative for decision-making.