Difference-in-Difference: A Versatile Method for Quasi-Experimental Evaluation

Significance. Social work researchers often use non-random quasi-experimental designs when it is illegal, unethical, or impractical to conduct randomized control trials. One of the most appealing of these is the difference-in-difference (DD) approach, an econometric method. DD simultaneously controls for differences between treatment conditions prior to treatment and change over time common to both groups, isolating the estimated treatment effect as change in the treatment group over time. DD can be implemented using regression, with variables representing Period, Treatment, and their interaction (Period x Treatment). The DD estimate is simply the coefficient of the interaction. DD can be extended to multiple periods and groups, and in certain settings the inclusion of selected covariates to control for confounding effects. DD is applicable to both cross-sectional and panel designs. In repeated cross-sections, participants in each treatment group at pretest can be different from the participants at posttest. In panel designs, the same participants are followed over time. In two-period (pretest-posttest) panel designs, analysis of covariance (ANCOVA) pretest-posttest regression is typically used, although DD is sometimes better. The critical assumption for DD estimates to be interpreted as causal effects is that in the absence of the treatment all groups would experience the same change, often called the "parallel trends" assumption. In a two-period design with one pretest, this is an untestable assumption. However, with 2 or more pretests, it is possible to obtain a pre-treatment estimate of change in the treatment group. Further, DD is compatible with a repeated measures multilevel design, a design familiar to many social work researchers for its ability to account for inherent growth or change. Content. This applied workshop will focus on the use of DD in non-experimental social work research settings. We will cover the basic framework for DD and compare it to ANCOVA in 2 period repeated cross sectional and panel designs. We will then cover repeated measures panel designs. A discussant will then describe real world examples of DD applied to social work research and evaluation. Recommendations for supporting the needed assumptions and addressing the typical challenges in each design will be provided. All participants will receive a handout detailing step-by-step DD procedures and best practices, and a list of DD resources. Implications. DD is an intuitive approach going back to John Snow's work identifying the source of the London cholera epidemic in the 1850s. It has lately achieved prominence as a rigorous econometric technique. Social work researchers should consider the potential uses for DD. First, it might be better in situations where they currently use ANCOVA. Both ANCOVA and DD require untestable assumptions, and the assumptions should be consonant with what is known about assignment to treatment. Second, they should design studies to have multiple pre-treatment observations so that change in the absence of treatment can be estimated to lend support to the parallel trends assumption. Third, DD facilitates pre-treatment control in certain repeated cross-section designs. Given these strengths, social work researchers can greatly benefit from knowing and using the DD method.