The Effects of Measurement Error in the Independent Variable On the Standardized Mean Difference and a Proposed Procedure for Correcting It

Background: Meta-analysis is becoming increasingly important for identifying evidence-based practices.  The standardized mean difference (SMD), one of the prominent effect sizes in meta-analysis, is used to represent the magnitude and direction of the difference between means of treatment and control populations in treatment efficacy research.  It is also used to represent the magnitude and direction of differences between means of populations of persons with different attributes, such as males and females, or persons who are and are not depressed.  In results not formerly in the literature, recent theoretical analysis has: (1) shown the population SMD as explicated in meta-analytic literature implies the absence of measurement error in an independent variable (IV) for classifying persons into different populations; and (2) derived a methodology for correcting the population SMD for measurement error in the IV.

Purpose: The consequences of measurement error in the IV for the population SMD and for the results of meta-analyses, and the efficacy of the proposed methodology for correcting the population SMD for measurement error in the IV, have yet to be examined in research.  This study addresses these gaps.

Methods: Simulation and modeling methodology was used.  In one simulation two populations of observed scores, true-SPi and true-SPj, on a dependent variable (DV) were simulated, with true-SPi a population of scores from persons who truly possess a characteristic of interest, i, say depression; and true-SPj a population of scores from persons who truly possess a different characteristic, j, say the absence of depression. Populations SPi and SPj were then formed by simulating classification of persons from true-SPi and true SPj into SPi and SPj based on scores from a hypothetical measure of the IV with varying amounts of measurement error.  The population SMDs for the differences between means of SPi and SPj were compared for scenarios involving differing levels of measurement error in the IV.  A second simulation used a parallel methodology but used data from a recent validity study of a suicidal ideation measure. 

Results: Results suggested measurement error in the IV significantly affects the population SMD, with its direction (algebraic sign) changing under realistic measurement error conditions.  In the simulation involving SPi and SPj, the population SMD ranged from +.50, when the IV was simulated measurement error free, to -.31 at the highest level of simulated measurement error.  The second simulation produced comparable results.  Use of the methodology for correcting the population SMD for measurement error in the IV demonstrated that under certain conditions the problematic effects of measurement error in the IV can be removed.

Implications: In realistic measurement error scenarios the population SMD estimated from a study to be included in a meta-analysis can have an algebraic sign opposite in direction from the “true” SMD.  The measurement error correction methodology can in some cases remove the negative effects of measurement error in the IV.  Implications of these results for conducting a meta-analysis, for consumers of published meta-analyses, and for future research on and theoretical development of the measurement error correction methodology are considered.