Findings and Implications for Evidence Building from a Cross-Site Implementation Study

Background and Purpose: In order to build capacity in evidence-based policy and program management, the Children’s Bureau Permanency Innovations Initiative (PII) purposively combined implementation science with evaluation research to strengthen child welfare evidence building. Six demonstration projects used the new structured PII approach to build replicable models to address the challenges of long-term foster care (LTFC). Implementation case studies chronicled project experiences at four successive evidence building phases: exploration, installation, initial implementation-formative evaluation, and full implementation-summative evaluation. A mixed-method case study protocol was used to answer three overarching questions:

• To what extent did projects complete evidence building phases/activities during the grant period?

• What facilitators and challenges were encountered during evidence building?

• What are the implications for future evidence building in child welfare?

Methods: The implementation study included case studies, an intervention fidelity tracker, a drivers assessment survey, an organizational readiness assessment, and analysis of the yearly grant expense reports. The mixed-method implementation case study protocol documented (a) the timing and duration of a comprehensive list of 54 activities spanning four evidence-building phases, (b) key products and accomplishments, and (c) challenges that delayed or compromised quality. Intervention fidelity was monitored through an Implementation Quotient (IQ) Tracker (Fixsen, Blase, Naoom, & Van Dyke, 2010). A drivers assessment survey elicited opinions on the status of supportive organizational policies and practices or “implementation drivers” (Kaye, Defanfilis, Bright, & Fisher, 2012). A onetime Web-based survey captured organizational readiness to implement evidence-informed models and rigorous evaluation. Grant expenditure reports were summarized by expenditure category for each fiscal year of funding.

Results: The case studies found that variations in starting child welfare contexts, governance structures, and legacy agendas among the grantees foretold their different journeys. As predicted, IQ Tracker scores showed that staff turnover had a negative effect on intervention fidelity. The drivers survey showed mixed results for most agencies over time with a mix of improvements and declines in driver status. On the organizational readiness survey, grantees tended to score the lowest on the measure of agency flexibility and ability to respond quickly to change, but all respondents agreed that evaluating the impact of programs is important and most were willing to use data collection instruments that required extra work.

Conclusions and Implications: A dearth of well-supported, child welfare relevant interventions posed a challenge to grantees, which required either adapting models from other fields or developing new models of intervention for addressing the local problems of long-term foster care. The decisions on interventions set the projects on different evidence-building paths owing to differences in the availability of existing core model components and proximal outcome measures. Grantees needed to spend more time examining expected causal links between their intervention and proximal and distal outcomes to ensure the usability of models in a child welfare setting before proceeding to initial implementation and formative evaluation; however, this process was challenged by a lack of time, available staff labor, and relevant research.