Session: Social Work Evaluations with Quasi-Experimental Designs: Challenges and Strategies (Society for Social Work and Research 20th Annual Conference - Grand Challenges for Social Work: Setting a Research Agenda for the Future)

Background and Purpose: Causal inference is a core interest of social work research. Although the randomized controlled trial (RCT) is recognized as the gold standard of research, true experimental designs are not always possible, practical, ethical, or even desirable. Given social work research’s reliance on quasi-experimental designs and the inherent challenges they pose in evaluation, researchers have sought methods to improve estimates of program effects. This symposium will discuss central challenges in evaluations with quasi-experimental designs and will call social work researchers’ attention to the importance of addressing selection bias in evaluations that lack randomization. Presenters will share emerging methods that enhance internal validity and enable researchers to address core evaluation issues efficiently.

The predominance of quasi-experimental designs in social work research is demonstrated in a recent review (Guo, 2014) of all articles published from January 1, 2012, to December 16, 2013, by five major social work journals: Social Work, Social Work Research, Research on Social Work Practice, Social Service Review, and the Journal of the Society for Social Work and Research. Among the 18 studies of program evaluation, six (33.3%) employed RCT designs at the individual level, two (11.1%) employed RCT designs at the group level, and 10 (55.6%) employed quasi-experimental designs. However, few studies used evaluation methods that address selection bias: Only 10 (55.6%) employed propensity score or other correction methods, and many of these methods were used to address the failure of the RCT in implementation.

Presentations: To promote awareness of the challenges in quasi-experimental design and the need to address selection bias in such studies, this symposium will provide an overview of methodological issues embedded in quasi-experimental studies that require analytic rigor and will summarize several new methods: propensity score modeling, an instrumental-variable approach, an interrupted time-series design, a regression-discontinuity design, and more. The symposium will present results from a federally funded study of the nation’s first rural Youth Violence Prevention Center (which serves an ethnically diverse rural county), illustrating use of an interrupted time-series analysis in evaluating the change of community-level youth violence indicators and in gaining supportive evidence on treatment effectiveness. To illustrate how a propensity score analysis of treatment dosage can overcome evaluation challenges, the symposium will discuss an evaluation of the effects of financial education on saving outcomes among low-income individuals who received Individual Development Accounts. The symposium will conclude with comments from two discussants (names are not shown here to ensure a blind review of the abstracts).

Implications: The symposium tackles fundamental challenges encountered in today’s social work evaluations, illustrates the importance of controlling for selection bias—the critical threat to internal validity of all evaluation projects, and discusses analytical strategies that are helpful for research by social workers and allied disciplines.