Formative Evaluation of An Evidence-Based Intervention to Reduce Long-Term Foster Care: Assessing Readiness for Rigorous Impact Evaluation
Methods: Randomized consent design (Zelen, 1990) was used to randomly assign 60 families of children with SED to treatment (PMTO) and control (foster care services-as-usual) conditions. Inputs and outputs were tracked as defined by the logic model. Outcomes data were collected in three domains: 1) child well-being, which included child behavior and mental health functioning; 2) family functioning, which included parenting skills, parental mental health, parental substance abuse, and readiness for reunification; and, 3) a permanency proxy (i.e., return home). Sources included worker assessments, caregiver self-report, and state administrative data. Descriptive and bivariate analyses were followed by examination of predetermined benchmarks for program inputs, enrollment, completion rates, fidelity, and intermediate outcomes, including effect size.
Results: Results were used to assess project performance in each category. Program inputs were fully installed as evidenced by staff selection, training, coaching, and performance assessment. Targets were met for enrollment (n=60) and completion rates (70%). Minimum fidelity requirements were attained by 91% of practitioners. Intermediate outcomes generally revealed improvements for the treatment group with statistically significant differences between treatment and control group on some but not all measures. The proxy for permanency was markedly better for the treatment group (return home: T=70%; C=40%), showing the program met the minimally desired effect size (OR=2.04). The odds of returning home was 3.5 times greater for the treatment group (OR=3.5, 85% CI=1.6-7.7).
Conclusion and Implications: Formative findings assessed implementation and intermediate outcomes in five areas, indicating the program met nearly all predetermined benchmarks. While not conclusive, findings suggest the intervention is associated with a permanency proxy. Compared to families receiving services-as-usual, treatment families’ children returned home at a notably higher rate. Findings indicate the program is ready for RIE. Moreover, this study contributes novel information about using an evaluation framework to examine formative findings and determine whether to progress to RIE. Researchers should consider using similar approaches to verify implementation and intermediate outcomes prior to investing in RIE. Implications for practice and policy also include using evaluation frameworks for systematic feedback to practitioners and decision-makers.