Methods for Conducting An Unbiased Meta-Analysis

Meta-analysis has become a preferred approach to doing research synthesis, and in a number of fields is now used as a quantitative approach to the identification of best evidence-based practices. Central to meta-analysis is the notion of an effect size, an indicator of the direction and magnitude of the results of a research study. The assumption has been that certain effect sizes, such as the correlation coefficient and standardized mean difference, place quantitative results onto a common metric so that outcomes from different studies based on different measures of the same construct can be meaningfully compared. Recent research by the presenter, however, has challenged this assumption, called into question a number of heretofore accepted meta-analytic methods, and led to recommended preliminary tests of measurement equivalence assumptions prior to doing a meta-analysis. These tests are of specific measurement equivalence assumptions that, if not met, may lead to erroneous meta-analytic conclusions and flawed identification of “best practices.” This workshop focuses on meta-analytic methods suggested by the results of this recent research. Participants will first go through a review of current meta-analysis procedures beginning with the identification of inclusion criteria and the literature search via systematic review, to the analysis of the accumulated effect sizes, and the interpretation and write-up of results. Then the focus will shift to a more specific emphasis on critical measurement equivalence requirements necessary for an unbiased meta-analysis, including how to develop a "validity map" that can be used to assess the plausibility that these equivalence conditions hold. The critical measurement equivalence assumptions necessary to an unbiased meta-analysis will be presented and discussed, the consequences of violations of these equivalence assumptions for the outcomes of a meta-analysis are considered, and specific tests of these equivalence assumptions described and illustrated. Participants will learn how-to-do a meta-analysis by being walked through an actual meta-analysis, from beginning to end, of research on the effects of participation in victim-offender mediation on subsequent delinquent behavior. Participants will learn how to frame questions appropriate for meta-analytic investigation; how to develop inclusion criteria; how to conduct a comprehensive and defensible literature review; how to choose an appropriate effect size statistic; how to conduct tests of important measurement equivalence assumptions that underlie meta-analysis; and how to do both fixed effects and random effects data analyses. The fixed effects and random effects analyses of aggregated effect sizes will be demonstrated using SPSS. Participants can expect to leave this workshop with basic knowledge necessary to conduct a meta-analysis of research in areas of interest. Participants will also have a basic understanding of critical measurement equivalence issues relevant for doing an unbiased meta-analysis, as well as knowledge of the biased results that can be obtained if these measurement equivalence assumptions are not met for the studies included in the meta-analysis. Participants will also be able to conduct specific tests of these critical measurement equivalence assumptions.