This paper presents results from a simulation study of discrepancies introduced into the results of meta-analyses when effect sizes are based upon measures violating CVI. Seventeen populations of true scores were created, modeling sixteen different studies comparing the effects of different treatments. The population true score SMD for each simulated study was computed based upon several different hypothetical measures, some of which met CVI and others which did not. The variability in results of meta-analytic analyses and comparisons of these effect sizes as a function of the measures violating CVI was investigated.
The results showed that the outcomes of meta-analyses of the effect sizes varied significantly, leading to contradictory conclusions about relative treatment effectiveness, as a function of violations of CVI. For example, in one analysis the relative ordering of effect sizes in terms of which showed more and which showed less effective treatments was completely reversed simply as a function of the measures the effect sizes were based upon when CVI was violated. Other results suggested that correlations between variables representing study characteristics and effect size magnitude, such as might be obtained in a moderator analysis, may change sign as a function of violations of CVI. Thus, for example, a given variable might be identified as being associated with smaller treatment outcomes when the effect sizes are based upon one set of measures, but then as associated with the largest of outcomes when the effect sizes are based upon a different set of measures. Proposed tests of CVI are presented and discussed. The implications of these results for the use of meta-analysis as a method for justifying evidence-based practices are considered.
Nugent, W. (2006). The comparability of the standardized mean difference effect size across different measures of the same construct: Measurement considerations. Educational and Psychological Measurement, 66(4), 612 – 623.
Nugent, W. (2009). Construct validity invariance and discrepancies in meta-analytic effect sizes based on different measures: A simulation study. Educational and Psychological Measurement, 69(1), 62 – 78.