Bridging Disciplinary Boundaries (January 11 - 14, 2007)

Purpose: Social work research has increasingly applied structural equation modeling (SEM). Making efforts to guarantee adequate power for SEM is especially important, because in SEM researchers hope to accept the null hypothesis about a model's overall fit; when this happens, it is crucial to ensure that the acceptance is not due to inadequate sample size. This study critically reviews statistical power of SEM commonly found in social work research, and makes recommendations for addressing challenges.

Methods: We first examined all articles in eight social-work or social-work-related journals published from January 1, 1999 to December 31, 2004. Of these journals, two (Social Work and Social Service Review) published zero article using SEM. The remaining six journals published a total of 139 articles using SEM: Journal of Gerontology, 51; Journal of Studies on Alcohol, 42; Social Work Research, 12; Research on Social Work Practice, 12; Child Abuse & Neglect, 12; and Journal of Social Service Research, 10.

We then evaluated statistical power for the SEM articles using the framework of power analysis for SEM developed by MacCallum and his colleagues (1996). Under this framework, one evaluates power based on effect size of RMSEA for close fit, for not-close fit, and for comparing nested models.

The two authors of this paper first evaluated these articles individually. They then compared review notes, discussed, and resolved disagreements.

Results: Of 139 articles, 32 or 23.02% do not have adequate power for testing overall model. Of these 32 articles, 7 or 22% do not have adequate power for testing nested models. Sample sizes for these studies range from 64 to 613, and degrees of freedom for testing the fit of overall model range from 2 to 57. The crucial factor affecting a study's power is not sample size alone, but sample size in relation to degree of freedom. A study with n=383 and df=20 leads to a marginal power of .746. The worst case is a study with n=110 and df=5, which only has a power of .134. For testing nested models, the worst case is a study with n=64 and “df difference”=2, which only has a power of .402. We investigated challenges threatening adequate power among these studies.

We have also found seven excellent studies that used small samples (ranging from 169 to 290) but relatively large degrees of freedom (ranging from 34 to 181). As a result, they all have adequate power.

Implications: Based on the above findings, we have made several recommendations for social work researchers to address the power issue. The most important implication to social work research is that researchers, reviewers, and journal editors should pay attention to statistical power of SEM, especially when they encounter studies with a small sample size in relation to a small degree of freedom.

References

MacCallum, R.C., Browne, M.W., & Sugawara, H.M. (1996). “Power analysis and determination of sample size for covariance structure modeling”, Psychological Methods 1(2): 130-149.