Formative Evaluation of An Evidence-Based Intervention to Reduce Long-Term Foster Care: Assessing Readiness for Rigorous Impact Evaluation

Purpose:  Despite distinct policy emphasis on permanency, many children do not attain permanency in a timely manner. One in five children in foster care has been in care for three years or longer (DHHS, 2012). The field faces an undersupply of evidence-based interventions to promote permanency, especially for children in long-term foster care (LTFC). In response, the Administration for Children and Families launched the Permanency Innovations Initiative (PII) in 2010. PII is a five-year, multi-site demonstration project designed to improve permanency outcomes by targeting specific groups of children at high-risk of LTFC. This study presents findings from one grantee’s local formative evaluation of an evidence-based parenting intervention – Parent Management Training Oregon Model (PMTO) – which targeted families of children with serious emotional disturbance (SED). Using Epstein & Klerman’s (2013) logic model approach to evaluation, this study builds knowledge about formative evaluation as a mechanism for assessing implementation and intermediate outcomes and determining whether an intervention is ready for rigorous impact evaluation (RIE).

Methods:  Randomized consent design (Zelen, 1990) was used to randomly assign 60 families of children with SED to treatment (PMTO) and control (foster care services-as-usual) conditions. Inputs and outputs were tracked as defined by the logic model. Outcomes data were collected in three domains: 1) child well-being, which included child behavior and mental health functioning; 2) family functioning, which included parenting skills, parental mental health, parental substance abuse, and readiness for reunification; and, 3) a permanency proxy (i.e., return home). Sources included worker assessments, caregiver self-report, and state administrative data. Descriptive and bivariate analyses were followed by examination of predetermined benchmarks for program inputs, enrollment, completion rates, fidelity, and intermediate outcomes, including effect size.

Results: Results were used to assess project performance in each category. Program inputs were fully installed as evidenced by staff selection, training, coaching, and performance assessment. Targets were met for enrollment (n=60) and completion rates (70%). Minimum fidelity requirements were attained by 91% of practitioners. Intermediate outcomes generally revealed improvements for the treatment group with statistically significant differences between treatment and control group on some but not all measures. The proxy for permanency was markedly better for the treatment group (return home: T=70%; C=40%), showing the program met the minimally desired effect size (OR=2.04). The odds of returning home was 3.5 times greater for the treatment group (OR=3.5, 85% CI=1.6-7.7).

Conclusion and Implications:  Formative findings assessed implementation and intermediate outcomes in five areas, indicating the program met nearly all predetermined benchmarks. While not conclusive, findings suggest the intervention is associated with a permanency proxy. Compared to families receiving services-as-usual, treatment families’ children returned home at a notably higher rate. Findings indicate the program is ready for RIE. Moreover, this study contributes novel information about using an evaluation framework to examine formative findings and determine whether to progress to RIE. Researchers should consider using similar approaches to verify implementation and intermediate outcomes prior to investing in RIE. Implications for practice and policy also include using evaluation frameworks for systematic feedback to practitioners and decision-makers.