Impact of Campbell Collaboration Systematic Reviews on Policy, Practice, and Research

Background/Purpose: There is a growing emphasis on evidence-informed decision making, and systematic reviews are seen as and increasingly important tool to assist policy-makers, practitioners and other stakeholders to make evidence-informed decisions. The Campbell Collaboration (C2) is an international not-for-profit organization that aims to produce, publish and disseminate systematic reviews of effects of interventions within the areas of social welfare, education, juvenile justice, and international development to improve decision making, bring about positive social change and improve quality of private and public service around the world. While it is acknowledged that C2 produces high quality rigorous reviews, it is unclear whether and how C2 reviews are being used. The aim of this study was to examine the impact of C2 reviews produced by the Social Welfare Coordinating Group (SWCG) on practice, policy, and research to inform future efforts to improve the utility and impact of C2 reviews.

Methods: We used a mixed method research design to examine impact of the 45 systematic reviews published by the C2 SWCG between January, 2003-December, 2014. An electronic survey was emailed to all corresponding authors of C2 SWCG reviews. Authors were asked 18 questions related to the publication and dissemination of C2 SWCG reviews and impact on practice, policy and research. We also examined impact by using data on downloads obtained from C2; conducted citation analysis using Google Scholar to capture citations of C2 reviews and other published version of the C2 review (e.g., journal articles); and used Altmetrics software to examine the extent to which C2 reviews are mentioned in social media, policy documents, newspapers, and blogs. Descriptive statistics were used to analyze the quantitative data and content analysis was used to analyze participants’ responses to open-ended questions. 

Results: A total of 45 systematic reviews have been published in the C2 SWCG library through December 2014. As of April 2015, the C2 reviews had been downloaded a total of 247,830 (m = 5,991) and cited 3,182 times. Twenty-three (51%) reviews received an Altmetric score of at least one, and 10 (22%) received a score of 10 or greater. The mean Altmetric score across all reviews was 9 (range 0-107). Twenty-four of the 35 corresponding authors (69%; some authors authored multiple reviews) participated in the author survey. Several authors provided additional evidence of reviews being used in practice and policy decisions and informing future research. 

Conclusions and Implications: C2 reviews are being accessed and used to inform policy and practice, although the extent to which they are impacting policy and practice is somewhat limited and varies by review. Systematic reviews can be an important resource for stakeholders as they synthesize, translate, and summarize a large amount of research, making it easier for stakeholders to digest and use. Producing high quality systematic reviews, however, is not sufficient; greater efforts are needed to increase the use and impact of C2 reviews to realize the ultimate aims of the Campbell Collaboration and increase the use of evidence to inform practice and policy.