Meta-Regression with Robust Variance Estimation: A Friendly Introduction with R

Systematic reviews and meta-analysis have become important scientific methods to inform evidence-based social work practice (Littell, & Maynard, 2015). Meta-analysis is a statistical procedure that quantitatively synthesizes empirical evidence across empirical studies. Over the past decade, meta-regression has emerged as a preferred meta-analytic procedure because (1) it is a multiple-regression like approach that is familiar to many social scientists; and (2) it allows flexible investigations on potential sources of heterogeneity (Chen, 2013). A complication increasingly encountered in meta-regression involves addressing dependent effect sizes that are reported in the same study. The dependence is created by a study measuring the same construct using multiple measures, or by measuring the same group of participants in multiple time points. This is problematic as it creates dependence among effect sizes, which violates the independence assumption of meta-regression. Existing ad hoc procedures for handling this dependence (e.g., aggregating effect sizes within each study or arbitrarily selecting one) compromise the validity of the meta-analytic inferences and restrict the meta-analytic research questions that can be investigated due to reduced number of effect sizes (sample size).

Multivariate procedures such as the use of generalized least squares (GLS) estimation have been encouraged as appropriate alternatives for handling the dependence (e.g., Becker, 2000; Gleser and Olkin, 2009; Riley, 2009; Raudenbush, Becker, & Kalaian, 1988). The GLS method requires information of the covariance matrices among pairs of effect sizes within studies. However, not all covariance formulae have been derived to address certain dependencies. In addition, the information needed to use the formulae is often not available in individual studies. Most recently, Hedges, Tipton, and Johnson (2010) have suggested and tested robust variance estimation (RVE) for meta-regression. RVE is currently a preferred approach for meta-regression due to its three advantages: (1) it does not require strong assumptions of multivariate normality; (2) it does not require exact values for covariances between dependent effect sizes, and (3) it is robust across both fixed- and random-effects assumptions. Workshop presenters will demonstrate the need for and the use of RVE for meta-analyzing dependent effect sizes. A high-level overview of the material will be provided that will include multiple meta-analytic examples. This workshop is designed to help participants: (1) understand sources and challenges of handling within-study dependence in effect sizes, (2) understand robust variance estimation and its robustness in addressing within-study dependence between effect sizes, (3) conduct meta-regression with Robust-Variance-Estimation using R software, and (4) learn to interpret RVE results reported in R software.

Career Level and Prerequisites: Participants are expected to have completed course work in meta-analysis or have conducted their own quantitative meta-analysis. Fundamental knowledge of univariate meta-analysis is preferred but not required.

Pedagogical Approach: The pedagogical approach will include a PowerPoint presentation, questions and answers, hands-on demonstration, and small-group activities. All data, codes, and materials will be shared with attendees.