Evidence, the Science of Improvement, and Program Evaluation: The Art and Science of Program Effects

Background:

Child protection practice in the United States is becoming more evidence-based. Federal funding for placement (foster care) prevention programs is tied to whether the public agency invests in evidence-based interventions. Without that evidence-based imprimatur, the cost of providing prevention services falls to state and local government, where historical inequities are likely to form. Though the policy is awkward, the intent is sensible. Child protection involvement is high-stakes for the families who touch the system. There is no reason to provide services that have no real track record, especially when child custody is at stake. That said, having the financial wherewithal to provide an evidence-based intervention to a family who needs the service is only one part of the problem government agencies face when making service investments they hope will improve outcomes. In this paper, from our experience working alongside public and private partners, we tackle more fully how evaluations are used as research evidence.

Methods:

Using Allegheny County as a case study, we develop a summative essay that describes the Hello Baby evaluation in the context of the research evidence use and improvement science literature. Specifically, we place the evaluation components (process and outcomes) into the plan, do, study, and act phases of the improvement cycle. For each phase, we describe the particular evidence needed and then show how the Hello Baby evaluation supplies that evidence. Moreover, evidence use is a behavior the depends on capability, opportunity, and motivation (COM). We align the actions in Allegheny with COM framework to show how the evaluation and other sources of evidence increased opportunity and capability in an already motivated county. Capability is defined as the knowledge and skills required for research evidence use. Motivation is defined as one’s desire and inclination to use research evidence to make job-related decisions. Opportunity refers to access to research evidence.

Results (Reflections):

There is a large body of work in health and education that tackles the problem of research evidence use when decisions are being made. Little of the work has found its way in child protection policy and practice research. There is an assumption that evidence use will have a positive effect, but there has been little research that ties all of this together in child protection: evidence that evidence use matters. The papers in this symposium and the summary essay show why the evidence use framework is critical to progress within social work research.

Conclusion

We are focused on these ideas because we think the implied questions and answers are connected to whether scale is achieved when social programs are implemented. The argument hinges on the idea that research evidence use will lead to better overall outcomes for the children and families who use those services. For that to happen, several dominoes have to fall into place in a rather orderly fashion. With Hello Baby, we show how one jurisdiction laid the ground for a deeper appreciation for how research evidence use improves outcomes.