Bridging Disciplinary Boundaries (January 11 - 14, 2007)

Meta-analysis is becoming a principal tool for systematic research reviews and for the identification of best practices not only in social work but also disciplines such as psychiatry and psychology. Fundamental in meta-analysis is the notion of an effect size, a parameter embodying information about the magnitude and direction of quantitative research outcomes. A basic tenet of meta-analysis is that effect sizes based upon different measures of a construct of interest are on a common metric and therefore directly comparable. Recent theory has contradicted accepted meta-analytic practice, suggesting that meta-analytic effect sizes based on different measures are, in fact, directly comparable only under limited circumstances. The purpose of this study was the investigation these recent theoretical claims concerning the standardized mean difference (SMD) and correlation effect sizes, and of the extent to which meta-analytic results vary as a function of measurement method. Three data sets were employed: one from a meta-analysis of the effects of victim-offender mediation (VOM) on re-offense; one from a study of dating violence (n = 120); and one from a study of comorbidity and suicidality (n = 406). The VOM meta-analysis was conducted under two different assumptions: that of direct comparability of effect sizes, and that of non-comparability. The other two data sets were combined and the reliability-corrected correlation effect size was computed for the relationship between depression and suicidality, and the reliability-corrected SMD for the difference in depression between two populations, based upon measures with differing metric properties. Results showed significant variability in effect sizes as a function of measurement procedure, consistent with theoretical expectation. The reliability-corrected correlation effect sizes ranged from .29 to .50, depending upon the relationship between the true score metrics of the involved measures. The reliability-corrected SMD effect sizes, for the same between-population comparison, ranged from -.09 to +1.05 as a function of the relationship between true score metrics of the involved measures. The raw score effect sizes varied even more substantially than did the reliability-corrected effect sizes. The results of the meta-analysis of victim-offender mediation studies differed as a function of whether or not effect sizes were treated as directly comparable. When treated as directly comparable, as in typical meta-analytic methodology, the results showed a time-decaying superiority of VOM in preventing re-offense as compared to traditional juvenile justice sanctions. When effect sizes were treated as non-comparable as a function of varying measurement metric properties, the results showed a superiority of VOM that varied as a function of measurement method and other methodological factors, but that was stable across time. These results imply that meta-analysis, as currently conducted, may be leading to erroneous conclusions as to best practices. The results suggest that social work researchers conducting systematic reviews should use meta-analysis only with extreme caution and with considerable attention to measurement issues. These results also suggest that, until these measurement issues are better understood, social workers may need to reconsider the use of meta-analysis as a principal tool for doing systematic research reviews.