A Review of Methods for Handling within-Study Dependence Among Effect Sizes in Quantitative Meta-Analyses

Background: Meta-analysis is used to summarize effect sizes from previous studies that have investigated a similar research question. Increasingly, primary studies report multiple effect sizes. If dependence among the multiple effect sizes is ignored or mishandled, then results of a meta-analysis can be problematic including lower power, less information, and potential under-estimation of standard errors for the synthesized effect size estimates and moderator coefficients. Instead within-study dependence is best handled using one of several choices of multivariate meta-analysis methods.         

There are various multivariate approaches that can be used, including generalized least squares (GLS) estimation, robust variance estimation (RVE), and the multilevel meta-analysis model. Use of GLS requires access to an accurate value for the correlation between pairs of effect sizes, but typically primary studies do not report all of the necessary information to calculate the correlation. The RVE method offers several advantages because its use does not depend as strongly on knowing the correlations between pairs of effect sizes within a study and does not require strict distributional assumptions. Use of the multilevel meta-analysis model also offers some similar advantages.

However, these models are complicated enough to require a large number of studies and effect sizes to work well. RVE can be used to handle within-study dependence for estimating a pooled effect size with as few as 10 studies. However, to estimate a meta-regression model that includes moderators, basic RVE estimation requires a minimum of 40 studies and, on average, five effect sizes per primary study. A small-sample bias correction version of RVE provides a good alternative under some scenarios for meta-analyses with smaller samples. To use the multilevel meta-analysis model, 30 studies are recommended. However, it is unclear that these criteria are well matched by the number of studies and effects that are commonly encountered in social science meta-analyses.

Methods: To assess the match between the sample sizes in applied meta-analyses and the sample size criteria for multivariate meta-analysis models, meta-analysis studies published between 2010 and 2016 in the Journal of Review of Educational Research were examined. Each meta-analysis was coded for the number of primary studies, average number of effect sizes, the types of analyses conducted and how within-study dependence was handled.

Results and Discussion: Of the 108 meta-analyses, 45 were quantitative meta-analyses. Most of the meta-analytic datasets entailed within-study dependence. The mean, median and mode of the number of primary studies per meta-analysis was 45, 35, and 22, respectively with a mean and median of the per-study number of effect sizes of 4 and 3. These sample size results indicate that a good proportion of applied meta-analyses are not large enough to satisfy sample size requirements for optimal handling of within-study dependence using multivariate meta-analysis models. Most of the meta-analyses employed ad hoc procedures to handle the within-study dependence. This paper will provide more detail about the kind of data encountered in real, applied social science meta-analyses, how meta-analyses are being conducted and offer recommendations for how to improve how within-study dependence in quantitative meta-analyses is handled.