Beyond Randomized Controlled Trials: Integrating Theoretical and Methodological Approaches to Causal Inference with Observational Data

Significance: In the realm of social work interventions generally, the what-works-question is increasingly important. In the specific case of child protection services, the federal government now gives funding priority to interventions designated as well-supported. To achieve the well-supported status, an intervention has to have been subjected to rigorous evaluation. In turn, rigorous evaluation is further defined in regulations, with strong, though not exclusive, preference given to randomized control trials (RCTs). The preference for RCTs is understandable; RCTs have long been regarded the "gold standard" for efficacy studies. That said, as an approach to the what-works-question, RCTs are not without their own problems, problems that include feasibility and challenges with external validity. Moreover, RCTs represent a rather narrow of view of how evidence is generated and knowledge is built. For example, the physical sciences have long relied on a combination of experimental and observational studies, motivated by theory, to build the knowledge base. In keeping with that well-worn path, the presentations in this workshop link theory with observational data in what-works studies of clinical efficacy.

Content: The workshop is organized in three parts. This workshop opens with a general discussion regarding the role of observational data, theory, and causal analysis in the what-works context. The discussion also considers how RCTs and observational studies are linked in the knowledge-building enterprise. The second and third parts focus on the practical application of these ideas. For theoretical orientation, we will introduce the concept of the gatekeeper in social work practice, and the implications of unmeasured gatekeeper selection tendencies for causal inference in non-experimental evaluation studies. To that end, we will introduce and describe the estimated best linear unbiased predictor (EBLUP) as a method for estimating latent gatekeeper effects, which has potential for broad application across social work research. We will demonstrate how to compute the EBLUP from mixed model estimates of treatment assignment, and discuss its benefits as a measurement approach. To demonstrate the benefits of the proposed approach, we will provide an example of using the EBLUP in a child welfare evaluation scenario using simulated data. We will follow that with the results from a real-world evaluation that used a random effects model to generate an adjusted estimate of worker referral tendencies. The discussion will highlight how random effects were then used to assess the efficacy of an intervention.

Implications: Answers to the what-works-question fit within the broader context of social work knowledge. The task of building that knowledge will, in the long-run, rely on a mix of evidence derived from both observational as well as experimental studies. The strength of evidence built will depend on how theory is used to generate questions that are then linked to methods best suited to the question rather than the other way around. In this workshop, participants will learn how theory and observational data come together to improve our understanding of treatment effects. Finally, because the methods are broadly applicable, implications for causal modeling over a broad range of social work issues will be addressed.