Integrating Formative Evaluation with Implementation Science in Complex Child Welfare Systems

Implementation Science has evolved to represent an important framework for developing the capacity of child welfare agencies to advance the effective installation and implementation of promising and evidence based practices. A major aspect of implementation science theory is the concept of implementation stages (Fixsen, Blase, Naoom & Duda, 2015). Based on the National Implementation Research Network (NIRN) framework Installation refers to the process of assembling the resources needed to build the foundation for and implement a practice, including formation of the implementation team, adjustments and interpretation of policy, development of manuals, and developing training and other resources identified through joint planning with the implementation team. As its name implies Initial Implementation is the process of starting to engage in the practice, using the new implementation processes, trying out the training, and identifying the aspects of the practice that may need to be further adapted in order for the practice to work as intended. This stage is described as the one that is most challenging.  To build evidence about what specific installation and initial implementation strategies work best, it is essential to simultaneously implement a formative evaluation that ensures that the intervention is stable and can be successfully implemented.

In order to improve outcomes for children and families, it is essential that child welfare systems and their stakeholders strengthen the knowledge base about how to successfully install and implement an intervention and determine whether these new practices work, for whom they are most and least effective, and how they can be consistently implemented in the real world. The presenters will describe the process of implementing a formative evaluation and applying implementation science in relation to two goals: 1) successful installation and initial implementation, and 2) the need to develop the evidence base for the practice.  

This workshop draws on the application of the installation and initial implementation stages in a large complex child welfare system--the New York City Administration for Children’s Services (ACS). Examples are based on the activities and adaptations needed to work through these implementation processes for an Enhanced Family Conferencing Initiative (EFCI). EFCI is designed for use during initial investigation when children are determined to be unsafe and at risk of out-of-home placement. Additionally, the workshop will expose participants to the challenges of developing evidence for the practice, which entails the use of a cluster random control trial.  In the complex ACS context achieving the goal of successful implementation in concert with the need to develop the evidence base as expected has proven to be extremely challenging.  Participants will be exposed to the components of the implementation stages and the general challenges associated with adapting the practice model and maintaining the integrity of a rigorous evaluation. These complex adaptive components include attention to CPS investigation policy, information confidentiality, implementation team development, staff development, organizational culture, internal and external communication, and the evaluation requirements. The presenters will facilitate a small group exercise to consider and discuss how these components interact and how competing requirements can be balanced.