The predominance of quasi-experimental designs in social work research is demonstrated in a recent review (Guo, 2014) of all articles published from January 1, 2012, to December 16, 2013, by five major social work journals: Social Work, Social Work Research, Research on Social Work Practice, Social Service Review, and the Journal of the Society for Social Work and Research. Among the 18 studies of program evaluation, six (33.3%) employed RCT designs at the individual level, two (11.1%) employed RCT designs at the group level, and 10 (55.6%) employed quasi-experimental designs. However, few studies used evaluation methods that address selection bias: Only 10 (55.6%) employed propensity score or other correction methods, and many of these methods were used to address the failure of the RCT in implementation.
Presentations: To promote awareness of the challenges in quasi-experimental design and the need to address selection bias in such studies, this symposium will provide an overview of methodological issues embedded in quasi-experimental studies that require analytic rigor and will summarize several new methods: propensity score modeling, an instrumental-variable approach, an interrupted time-series design, a regression-discontinuity design, and more. The symposium will present results from a federally funded study of the nation’s first rural Youth Violence Prevention Center (which serves an ethnically diverse rural county), illustrating use of an interrupted time-series analysis in evaluating the change of community-level youth violence indicators and in gaining supportive evidence on treatment effectiveness. To illustrate how a propensity score analysis of treatment dosage can overcome evaluation challenges, the symposium will discuss an evaluation of the effects of financial education on saving outcomes among low-income individuals who received Individual Development Accounts. The symposium will conclude with comments from two discussants (names are not shown here to ensure a blind review of the abstracts).
Implications: The symposium tackles fundamental challenges encountered in today’s social work evaluations, illustrates the importance of controlling for selection bias—the critical threat to internal validity of all evaluation projects, and discusses analytical strategies that are helpful for research by social workers and allied disciplines.