Advanced Meta-Analysis: Handling Multiple Effect Sizes per Study Using Robust Variance Estimation

Systematic reviews and meta-analysis can be used to synthesize results across studies to provide stronger empirical evidence that can inform evidence-base social work practice (Littell, & Maynard, 2015). A complication increasingly encountered in meta-analysis involves addressing a scenario in which individual studies report multiple, related effect sizes. The resulting dependence most commonly originates from intervention effects (and thus effect sizes) being reported for multiple related outcomes and from multiple treatments being compared with a common comparison group. Ad hoc procedures for handling this dependence (e.g., aggregating effect sizes within each study or ignoring the dependence) compromise the validity of the meta-analytic inferences and restrict the meta-analytic research questions that can be investigated.

Instead, multivariate procedures including the use of generalized least squares (GLS) estimation have been encouraged as appropriate methods for handling the dependence (e.g., Becker, 2000; Gleser and Olkin, 2009; Riley, 2009; Raudenbush, Becker, & Kalaian, 1988). However, use of GLS requires knowledge of the covariances among pairs of effect sizes within studies and while researchers have derived formulas for certain specific dependencies, not all covariance formulas have been derived. More importantly, the information needed to use the formulas is not always reported in primary studies. Last, use of GLS also depends on an assumption of multivariate normality.

Most recently, Hedges, Tipton and Johnson (2010) have suggested and tested use of robust variance estimation (RVE) for multivariate meta-analysis. RVE does not require strong assumptions of multivariate normality and does not require exact values for covariances between dependent effect sizes.  

Workshop presenters will explain the need for and demonstrate the use of RVE for meta-analyzing dependent effect sizes. A high-level overview of the material will be provided that will include multiple meta-analytic examples. This 90-minute workshop is designed to help participants: 1.Understand sources and implications of ignoring within-study dependence in effect sizes encountered by meta-analysts; 2.Understand robust variance estimation and how it can be used to handle within-study dependence between effect sizes; 3.Learn how to use an SPSS macro that implements RVE to estimate random effects meta-analytic models to calculate pooled effect size estimates and to test hypotheses about moderators; 4.Learn how to interpret RVE results reported in SPSS output when estimating random, and mixed-effects meta-regression models

Career Level and Prerequisites:

With an emphasis on addressing a common methodological dilemma when conducting a meta-analysis, this workshop is appropriate for SSWR attendees who have completed course work in or who have conducted their own quantitative meta-analysis. Fundamental knowledge of univariate meta-analysis is assumed.

Pedagogical Approach:

The pedagogical approach will include: a PowerPoint presentation with handouts, lecture interspersed with questions and answers and discussion, software demonstration and small group work focused on interpreting results. Time permitting, we will include small group exercises for attendees to run analyses themselves using SPSS. All data, code and overheads will be shared with attendees.