Understanding Practice Change across an Evaluation of the Child Welfare Capacity Building Collaborative

Purpose and Background

Technical assistance (TA) and training remain integral components of child welfare systems across the country (Barbee, et al., 2019). Yet, evaluating technical assistance efforts remains an ongoing challenge, especially those focused on practice change. In partnership with the Children’s Bureau, this presentation explores a subset of findings from a five-year longitudinal evaluation of the Child Welfare Capacity Building Collaborative (CBC). Consisting of three centers, CBC aids courts, states, and tribal child welfare agencies in facilitating positive outcomes in child and family well-being (James Bell Associates and ICF, 2020; 2022).

Evaluators examined how the centers impacted practice change outcomes at the jurisdictional levels guided by the following questions:

• To what degree do jurisdictions perceive that agency/court practice is changing as a result of Center capacity building services?
• Additionally, what are the core components of practice change within the CBC?

Methods

Over 5 years, researchers combined purposive and convenience sampling to collect data on perceived practice changes. Data originated from an outcome practice questionnaire for jurisdiction staff (n=72), TA reported practice change outcomes from a project-tracking database (n= 172), and interviews with jurisdiction leaders (n= 97).

Researchers blended Interpretive Phenomenological Analysis and content analysis to explore practice change across the centers. In the first phase, two coders open coded all data independently. In the second phase, independent coding was compared to identify shared patterns, reconcile differences, and finalize coding schemes.

Results

Across data sets, three core categories, multiple subcategories, and patterns emerged. The dominant category, Complications in Practice Change (CPC), covered barriers that hindered or prevented change. The second core category, Organizational Systems and Infrastructure (OSI), covered planning and development, data and information systems, new programs, direct practice, and leadership. The third core category, Community Engagement (CE), involved building partnerships as a practice change outcome.

Like OSI, CE responses linked to CPC in complex ways. While most CPC responses explicitly indicated they perceived no practice change, OSI and CE subcategories closely align with CPC’s subcategories shown above.

Discussion and Implications

Findings demonstrate CPC, OSI, and CE broadly capture the degrees of practice change-focused technical assistance within the CBC. However, jurisdictions varied on what they perceived as practice change. These findings have implications for practitioners, evaluators, and child welfare scholars. First, the core categories, dimensions, and patterns identified can act as an early stage set of practical guides for TA and evaluation adaptable to various local contexts (see Bollier, & Helfrich, 2019). Second, findings highlight the importance of developing shared understanding of practice change to effectively develop, implement, and evaluate technical assistance efforts. Finally, findings provide key insights for assessing CBC’s collective impact and building evidence informed TA approaches.