A Monte Carlo Study of Variability in the Results of Meta-Analyses As a Consequence of Including Non-Comparable Measures in the Meta-Analyses

Meta-analysis is an important tool for identifying evidence based practices.  A fundamental tenet of meta-analysis is the comparability of effect sizes, such as the standardized mean difference (SMD), based on different measures.  Recent research has suggested SMDs based on different measures are, in fact, comparable only if the measures are congeneric, hence measure the exact same construct.  Recent research has suggested SMDs representing the difference between two given populations, but based on scores from non-congeneric measures, may differ considerably.  These findings implied the results of a meta-analysis may differ between the circumstance in which the included studies all use congeneric measures and the circumstance in which the studies use non-congeneric measures.  No study has yet tested this possibility.  This possibility raises potential challenges to the validity of meta-analytic results for identifying and justifying evidence based practices. 

This Monte-Carlo study investigated the extent to which the results of a meta-analysis varied as a function of congeneric and non-congeneric measures being used in the included studies.  The study was an investigation of the contrapositive:  How would the results of a meta-analysis have differed had the true score SMDs analyzed in the meta-analysis been based on scores from different combinations of congeneric and non-congeneric measures? 

Six populations of true scores were simulated, five representing populations each of which had received one of five interventions, and one representing a no-treatment control population.  The populations were comprised of true scores from seven hypothetical measures A, B, C, D, E, F, and G, only two of which were congeneric.  The relationships between the true scores from hypothetical measures A thru G were modeled to be either congeneric, unidimensional non-linear, or two-factor.  Six-thousand random samples of scores from the six populations were obtained, simulating 6,000 “studies,” and 42,000 true score SMDs, for comparing the effects of the five hypothetical treatments.  A random sample of these “studies” was drawn and fixed effects analyses of the studies’ SMDs conducted.  Multiple analyses of the sampled studies’ SMDs were conducted based on scores from different combinations of the hypothetical measures.  This facilitated comparisons of meta-analytic results when based on scores from different measures, allowing investigation of questions of the form, “How would the results of the meta-analysis have differed had the true score SMDs been based on scores from measures A, B, D, F, and G instead of all having been based on scores from measure A?”.

Findings showed results of tests of homogeneity of SMDs varied substantially.  This suggested conclusions regarding homogeneity of SMDs can be affected by the particular combinations of measures used in the included studies.  Findings showed statistically significant differences in the rank ordering of the magnitude of SMDs (i.e., the rank ordering of effectiveness of treatments) as a function of different combinations of congeneric and non-congeneric measures.  These findings suggested that meta-analytic results may be sensitive to the measures used in the included studies.  These results suggest the need for cautious interpretation of meta-analytic results and the need for further research in this area.