Friday, 14 January 2005 - 8:00 AM

This presentation is part of: Methods of Research Synthesis: Providing Credible Evidence for Practice and Policy

Methods of Research Synthesis

Reviewers use various methods to identify, assess, and synthesize empirical evidence. Even so-called “systematic reviews” vary in the extent to which they use explicit inclusion and exclusion criteria, well-defined search and retrieval procedures, and state-of-the-art analytic methods.

Purpose: The purpose of this paper is to provide an overview of popular and emerging methods of research synthesis used to summarize effects of interventions. Emphasis is on potential effects of decisions in the review process on conclusions about the evidentiary status and effectiveness of interventions.

Methods: To achieve this purpose, commonly used review practices, as revealed in 25 reviews of studies of Multisystemic Therapy, are compared with international, interdisciplinary guidelines for systematic reviews developed by the Cochrane and Campbell collaborations. The results of this comparison are then considered in relation to empirical research on the impact of different inclusion criteria, search strategies, methods of assessing study quality, and methods of research synthesis on conclusions regarding the quality of available evidence and effectiveness of an intervention.

Results: This analysis shows that research reviews are likely to be biased if they are based on: convenience samples of studies rather than systematic searches; recent, published reports (versus all reports on relevant studies); statistical significance of study findings rather than effect sizes; and pooled effect size estimates that are not weighted by study quality and precision. Stronger reviews use clear a priori inclusion and exclusion criteria; well-specified search procedures that include extensive efforts to find relevant unpublished studies; careful analysis of study design and implementation, intervention characteristics, counterfactual conditions, population and setting characteristics, and outcome measures; and meta-analysis when possible.

Implications for practice and policy: Systematic reviews can provide strong support for empirically-based practice and policy, but review methods matter. Guidelines for systematic reviews have been developed through international, interdisciplinary collaboration among scholars in order to improve the field of research synthesis. These guidelines can be incorporated in doctoral student training and professional education.

See more of Methods of Research Synthesis: Providing Credible Evidence for Practice and Policy
See more of Symposium

See more of Celebrating a Decade of SSWR (January 13 - 16, 2005)