Methods: Applying a two-group quasi-experimental design with repeated measures, data were collected at baseline, 2-month follow-up, 1-year follow-up, and 2-year follow-up using a web-based survey. Among those participating (N=590; intervention=372; comparison=218), 54% were child welfare caseworkers, 19% were child welfare supervisors, and 27% were community partners. Measures included self-reported preparedness and ACSCA practice behaviors. To address study attrition due to worker turnover and survey non-response, data were imputed using multiple imputation. Using intent-to-treat, multivariate analyses were conducted using mixed linear models that examined if the intervention and comparison sites differed on: (1) ACSCA practice behaviors increasing over time, controlling for level of preparedness, provider role, and project site, and (2) level of preparedness to engaging in ACSCA practice behaviors increasing over time.
Findings: At baseline, we observed no significant difference between intervention and comparison groups on outcome measures. Intervention exposure was not directly related to practice behaviors; however, preparation was positively related to practice behaviors (b(SE)=0.39(0.06), p<0.001). For how well-prepared respondents felt to engage in the ACSCA, we observed a time x intervention interaction where intervention group preparation was higher than comparison group preparation during Time 3 and Time 4. For the intervention group, post-estimation test indicated significant within group differences between T1 and T4 (F(1,577.5)=6.12, p=0.014) and T1 and T3 (F(1,518.5)=15.63, p<0.001). Preparedness to engage in practice behaviors likely mediates the relationship between intervention exposure and practice behaviors.
Conclusion/Implications: Within a 2-year period, we observed that intervention activities likely only indirectly contributed to practitioners’ competence in engaging in ACSCA practice behaviors; the main factor that likely promotes application of the ACSCA is practitioners’ confidence in their ability to engage in these practices. Worker confidence took time to develop through intensive training and coaching; yet, after two years, their confidence had not yet translated to differential competence. These findings are supported by implementation literature that suggests that high-fidelity, full implementation may require 2-4 years and that confidence precedes competence (Fixsen, Blase, & Van Dyke, 2019). It is also noteworthy that the implementation phase took place during the COVID-19 pandemic. As such, even more time may have been required for multi-level, large-scale, sustained change. Results suggest that extended time periods, greater than two years, would be better suited to observe change in large-scale evaluation studies.