Building Capacity of Child Welfare Agencies to Use Evidence Supported Interventions

Background and Purpose: Compared to related fields, such as education, mental health, criminal justice, and public health, child welfare has historically lagged behind in building evidence for what works to ensure the safety, permanence, and well-being of children in foster care. In 2010, only 20 (9%) of the 223 programs catalogued by the California Evidence-Based Clearinghouse for Child Welfare were well-supported by research evidence. As of April 2017, this number has risen to only 29 (7%) out of 401 programs, only two of which were designed or commonly used for children served by the child welfare system. In proportionate terms, the field is losing ground.

The PII Approach is based on the premise that an innovation must pass through various “tollgates” of implementation integrity and intervention validity to build evidence of its efficacy and effectiveness before it can be sustained, adapted, and replicated. The phases of the PII Approach of implementation validity and intervention validity tollgates can help create quicker, cheaper, and ultimately smarter ways of building the evidence base for what works in child welfare.

Methods: The PII Approach begins with an Exploration and Installation stage. During this phase, the needs of the target population and potential interventions are “explored” to determine how best to address an agency’s specific challenges related to long-term foster care. The intervention is then “installed,” to monitor how services are delivered and measured. The processes of delivering the intervention and collecting data are examined through “usability testing” with a handful of subjects. Short-term outcomes are subsequently assessed on a larger sample through a formative evaluation. For those innovations that show promise by demonstrating a statistical association between the intervention and program outputs and proximal outcomes, which trend toward significance (e.g., P < .15), they advance to Full Implementation and Summative Evaluation. Results from this phase tell us whether long-term outcomes are achieved, on average, and the extent to which these outcomes can be attributed to the causal effects of the intervention.

Results: Two projects that adopted proven models from related fields reached full implementation and completed randomized controlled trial (RCT) summative evaluations. A third project combined existing promising child welfare models and also completed a RCT summative evaluation. Two developmental projects completed two rounds of formative evaluation, but did not have time to progress to full implementation and summative evaluation. The site-specific evaluations found mixed results among the projects on both proximal and distal outcomes.

Conclusions and Implications: Conventional methods of rigorous evaluation design maximize internal validity at the expense of external validity, which do not always translate well to the practical realities of working with mostly involuntary populations of child welfare clients. Future evaluations seeking to build evidence for what works in child welfare should apply more stringent toll-gate criteria for usability testing and formative evaluation to prevent ineffective interventions or poorly implemented models from reaching full implementation and strengthen on-going operating measurement of the enabling context (e.g., caseload size, staff turnover) at both formative and summative evaluation.