Why One Method Won't Work: A Mixed-Method, Longitudinal Approach to Evaluating Training and Technical Assistance (T/TA)

Background: The cross-site evaluation of Implementation Centers (ICs) and National Resource Centers (NRCs) utilized a mixed-method approach to evaluate T/TA provided to child welfare systems. The evaluation addressed barriers and facilitators to T/TA usage; T/TA quality; nature of providers’ relationships with jurisdictions; and outcomes of T/TA.  Evaluation methods were integrated and data were triangulated to comprehensively evaluate T/TA.

Methods: In-depth, semi-structured interviews were conducted with child welfare directors from 50 States, DC and Puerto Rico and a sample of 22 directors from Tribes/Territories receiving title IV-B funding. Interviews were conducted three times in 18-month intervals with response rates of 77%, 80%, and 81%. Interviews captured directors’ perceptions of T/TA utilization, quality, relationships with providers, the child welfare systems’ organizational and systems changes, and the contributing role of T/TA. Web-based Satisfaction Surveys were administered semi-annually to a sample of T/TA recipients (stratified by provider and T/TA location). Surveys gathered information on the quality and coordination of T/TA and recipients’ relationships with providers. 708 surveys were disseminated and 267 responses were received (38%). Five Longitudinal Case Studies were conducted in jurisdictions receiving IC and NRC T/TA. Annual data collection over three years involved document review, observations, and interviews with agency staff, stakeholders, T/TA providers, and Federal staff. Case studies assessed how context influenced T/TA’s impact. Annual Semi-structured Interviews with T/TA Providers and Federal Staff were conducted; topics included provider relationships with clients, and facilitators and barriers to providing T/TA and to achieving desired outcomes. Document Review: NRC final evaluation reports and IP Reports were reviewed to identify outcomes of T/TA.

Interview data were analyzed using qualitative methods, using ATLAS.ti to organize information and identify themes and patterns in the data. Scaled items were coded and entered into SPSS and analyzed using quantitative methods.

Results: T/TA utilization was facilitated by the development of jurisdictions’ program improvement plans and prior relationships with providers, but hindered by staff’s limited availability to engage with providers. Overall, recipients were satisfied with the T/TA quality and reported positive interactions with providers. Of the 312 capacity building and systems changes reported by child welfare directors, Children’s Bureau’s T/TA providers were cited as contributing to 45% of these changes. Most frequently reported capacity building changes included managing with data; building leadership capacity, and implementing family finding and quality assurance systems. Top cited systems changes were addressing out-of-home care, improving safety, and implementing practice models, Facilitators to achieving change included length of tenure of leadership and senior administrators; leadership’s involvement in the change initiative; and agency’s organizational culture. Barriers included lack of financial and staffing resources, and staff turnover.

Implications: T/TA supports capacity building and system change efforts in State/Tribal child welfare systems. It is best facilitated when T/TA is aligned with appropriately identified needs, leaders and key stakeholders are actively engaged, and jurisdictions provide staff to shepherd the initiative. T/TA must support jurisdiction’s knowledge of implementation processes.