Differences in Meta-Analytic Results As a Consequence of Measurement

Background and Purpose:  Recent standardized mean difference (SMD) effect size theory has postulated SMDs, for a given population comparison, based on different measures are comparable in the sense of being completely interchangeable only if the measures are equivalent in terms of validity and reliability.  SMDs for a given study based on scores from measures i and j are completely interchangeable if they can be interchanged in a distribution of SMDs with no changes in statistical or conceptual meaning.  This has led to the following conjecture: Suppose the SMDs for n studies in a meta-analysis were each based on one of the measures from a set of measures A, B, and C. If the SMD in each study were instead based on a measure from a different set of measures D, E, and F, and the SMDs based on the different measures were not completely interchangeable, the results of a second meta-analysis of the SMDs could differ significantly from the first. Model based and Monte Carlo simulations have produced results consistent with this hypothesis.  This study tested this conjecture in the context of an actual meta-analysis. 

Method:  Secondary meta-analyses of the 14 studies included in a recently published meta-analysis of treatments for depression in older adults were conducted.  Twelve of the 14 studies used multiple measures of depression.  Secondary meta-analyses were conducted, each with an included study’s SMD based on a different measure.  This allowed a test of the above conjecture.   

Results:  The original meta-analysis found no difference between outcomes of cognitive-behavior therapy (CBT) and reminiscence therapy (RT) for depression.  Results of secondary meta-analyses differed to both statistically and practically significant degrees.  Results showed when SMDs in the included studies were based on one particular combination of measures, the mean SMD for CBT was -.39 (95% confidence interval [CI], -.67 to -.10), and that for RT, -.55 (95% CI, -.83 to -.27), a statistically non-significant difference.  But with SMDs based on a different set of measures the mean SMD for CBT was, -1.28 (95% CI, -1.57 to -.99), and that for RT was, -.47 (95% CI, -.75 to -.19), a statistically and practically significant difference.  One meta-analysis suggested no meaningful difference between the outcomes for CBT and RT, yet the second suggested CBT superior to RT, with a difference of -.81 between mean SMDs. 

Conclusions and Implications:  These findings were consistent with the hypothesis that when SMDs based on different measures are not comparable in the sense of being completely interchangeable, the results of a meta-analysis can depend on which measures happen to be used in which studies.  In at least some cases meta-analytic results may be an artifact of measurement.  These findings raise cautions about meta-analysis as a tool for identifying evidence-based practices.  These findings suggest: (1) an approach to assessing the sensitivity of meta-analysis results to problems with non-comparability of SMDs based on different measures, and (2) further research is needed on the conditions under which lack of complete interchangeability of SMDs in a meta-analysis lead to artifactual findings.