Friday, 14 January 2005 - 8:00 AM

This presentation is part of: Methods of Research Synthesis: Providing Credible Evidence for Practice and Policy

Evaluating a Body of Evidence

Guidelines for evaluating the weight of empirical evidence across multiple studies have been developed by meta-analysts (Cooper & Hedges, 1994; Lipsey & Wilson, 2001; Valentine & Cooper, 2003) and others (e.g., Chambless et al., 1998). These guidelines include both methodological and practical considerations, although they vary in emphasis and level of specificity. Previous systematic reviews (Littell, 2004; Scher, 2003) indicate that different approaches to evaluating a body of evidence lead to different conclusions.

Purpose: This paper examines and applies prominent and popular schema for evaluating a body of evidence.

Methods: The Cumulative Research Evidence Assessment Device (CREAD, Valentine & Cooper, 2003), Chambless APA criteria (Chambless et al., 1998), and various meta-analytic approaches were applied to a set of eight controlled studies of Multisystemic Therapy (MST).

Results: Conclusions about the effectiveness of an intervention (e.g., MST) depend, in part, on which criteria are utilized to evaluate the body of evidence. The Chambless criteria are the easiest to use, least informative, and potentially misleading. The CREAD is complex, but offers some advantages over traditional meta-analysis. Conclusions based on meta-analysis depend, in part, on earlier decision rules (regarding inclusion criteria, the search strategy, study-quality assessment, etc.) and whether and how researcher handles variations across studies in terms of their methodological quality, interventions, counterfactual conditions, populations, and setting characteristics.

Implications for practice and policy: The most commonly-used approaches to evaluating a body of evidence on the effects of a social intervention are not the most rigorous or reliable. Implications for further research, training, and publication policies are considered.

See more of Methods of Research Synthesis: Providing Credible Evidence for Practice and Policy
See more of Symposium

See more of Celebrating a Decade of SSWR (January 13 - 16, 2005)