When Is Small Large Enough?: An Empirical Investigation on the Impact of Sample Size and Model Complexity on Two-Level Linear Models

Background: Multilevel models (MLM or hierarchical linear models and mixed models) are increasingly used in social work research to analyze nested or hierarchically structured data (e.g., students nested within schools). As their use expands, questions have emerged concerning how well these models work under various conditions. Given the often small samples used in social work research, one critical condition is the sample size at each level of the analysis. Whereas vague guidelines have been proposed over the years [e.g., 30/30 rule which states that in order to estimate linear multilevel models, a study needs at least 30 level-2 units (e.g., school) and 30 level-1 units (e.g., children)], Much is still to be learned regarding small samples and MLMs. For example, what do results reveal from models with 10 or 15 level-2 units and 100 or more level-1 units? Are models with small level-2 sample sizes better or worse than single-level contextual models? This study aims to address such lingering questions. Specifically, by examining a variety of sample size combinations and more complex multilevel models (i.e., two-level models with various numbers of predictors, various levels of collinearity, and binary and continuous predictors at each level), this study adds information about the accuracy and precision of estimates and contributes to our understanding of the behavior of multilevel models under data conditions that are often encountered in social work research. 

Methods To help generate MLM sample size guidelines for applied social work researchers, Monte Carlo simulation methods were used to examine how level-1 sample size, level-2 sample size, intercept and slope variance, collinearity, and model complexity impact model convergence rates, parameter estimates, Type I error control, and statistical power of tests associated with the fixed effects from two-level models and contextual, single-level OLS models.

Results: When the nested data structure was ignored and single-level OLS models were estimated, regardless of sample size, level-2 variables were extremely biased.  Likewise, the Type I error rates for the OLS models ranged from 0.3 to 0.6.  However, on average, bias and Type I error rates from the MLMs were within acceptable ranges, regardless of sample sizes. Findings for statistical power were mixed. Whereas level-1 variables, on average, had acceptable levels of statistical power (except in the extremely small sample sizes), level-2 variables rarely exhibited adequate levels of statistical power.

Implications: Small samples are a reality of social work research.  However, given the results from this comprehensive simulation study, social work researchers need not shy away from using MLMs to answer their research questions.  Instead, researchers can feel confident using MLM with small samples. Although MLMs with small sample sizes might be underpowered, overall, results from such models are not biased and Type I error rates are not inflated. Thus, social work researchers that investigate relationships between social contexts (e.g., schools, neighborhoods) and individual outcomes should consider using MLM in future research.