Doing a Meta-Analysis Right

Meta-analysis has become a preferred approach to doing research synthesis, and in a number of fields is now used as a quantitative approach to the identification of best evidence based practices. Central to meta-analysis is the notion of an effect size, an indicator of the direction and magnitude of the results of a research study. The assumption has been that certain effect sizes, such as the correlation coefficient and standardized mean difference, place quantitative results onto a common metric so that outcomes from different studies based on different measures of the same construct can be directly and meaningfully compared. Recent research by the presenter, however, has challenged this assumption, called into question a number of heretofore accepted meta-analytic methods, and led to recommended preliminary tests of measurement assumptions prior to doing any meta-analysis. These tests are of specific measurement assumptions that, if not met, may lead to erroneous meta-analytic conclusions and flawed “best practices.”

This workshop focuses on “how to do a meta-analysis” based upon the results of this recent research. Participants will learn how to do a meta-analysis, from the identification of inclusion criteria and the literature search, to the analysis of the accumulated effect sizes, and the interpretation and write-up of results. Central to this over view will be discussion of critical measurement issues relevant for an unbiased meta-analysis, including specific tests that need to be conducted prior to doing a meta-analysis. The critical measurement assumptions necessary to an unbiased meta-analysis will be presented and discussed, and specific tests of these assumptions described and illustrated.

Participants will learn how-to-do a meta-analysis by being walked through an actual meta-analysis, from beginning to end, of research on the effects of participation in victim-offender mediation on subsequent delinquent behavior. Participants will learn how to frame questions appropriate for meta-analytic investigation; how to develop inclusion criteria; how to conduct a comprehensive and defensible literature review; how to chose an appropriate effect size statistic; how to conduct tests of important measurement assumptions that underlie meta-analysis; how to identify and code independent variables; how to do both fixed effects and random effects data analyses. The fixed effects and random effects analyses of aggregated effect sizes will be demonstrated using SPSS.

Participants can expect to leave this workshop with the basic knowledge necessary to conduct a meta-analysis of research in areas of interest. Participants will also have a basic understanding of critical measurement issues relevant for doing a meta-analysis, as well as knowledge of the biased results that can be obtained if specific measurement assumptions are not met for the studies included in the meta-analysis. Participants will also be able to conduct specific tests of these critical measurement assumptions.