Measurement Equivalence, Symmetry, Effect Sizes, and Meta-Analysis

Background and purpose:  Meta-analysis is at the top of many evidence-based hierarchies.  The number of meta-analyses published in social work and related professions are increasing.  Fundamental to meta-analysis is the presumption that effect sizes (ESs), such as the standardized mean difference (SMD), based on scores from different measures are on the same metric and comparable.  Recent argument and simulation studies challenge this presumption.  A challenge to this premise is that research has shown SMDs based on scores from different measures are comparable only if the scores from the different measures meet a form of measurement equivalence expressed by two criteria, (1) construct validity equivalence and (2) equal reliabilities.  A problem with this challenge is it appears ad hoc, with no theoretical underpinning.  This ad hoc nature limits the extent to which this argument concerning ES comparability can be used to guide meta-analysis methodology and be extended to ESs other than the SMD.  The current research addresses this inadequacy.

Methods:  A model of measurement equivalence is developed in which validity is manifested by certain symmetries seen in distributions of true scores from different measures, and in the degree of approximate symmetries observed in distributions of observed scores from the different measures.  The invariance, hence comparability, of certain ESs based on different measures emerges as a consequence of these symmetries and approximate symmetries.  This model suggests two classes of invariant ESs.  To examine the predicted invariances, simulation studies are conducted to test hypotheses about ES invariance across different measures.  These simulations employ both model based and Monte Carlo methodologies.  Not only is ES invariance and comparability tested, the degree to which ESs for a given “effect” diverge as a consequence of degree of broken symmetry, with degree of broken symmetry indicated by the correlation between scores from the different measures, is also examined.

Results:  The results of these simulations are consistent with model predictions.  Results show ESs such as the SMD, correlation, odds ratio, relative risk ratio, common language ES, and distribution overlap ES are invariant when the symmetries associated with conditions (1) and (2) hold.  Results also show that the greater the degree of broken symmetry, as indicated by the correlation between scores from the different measures decreasing increasingly from 1.0, the greater the difference found between ESs for the same “effect” when based on scores from the different measures.

Conclusions and implications: This research suggests, first, that further theoretical research to identify other possibly invariant ESs when the symmetries implied by conditions (1) and (2) hold.  This may lead to the identification of new, currently undefined invariant ESs.  Second, the results also suggest possible tests for the symmetries underlying the invariant ESs that might be used as a preliminary step in a meta-analysis.  Third, the results also suggest a number of avenues of future research investigating the degree to which ESs vary under conditions of broken symmetry associated with violations of measurement equivalence as defined by conditions (1) and (2).