Examining Implementation and Program Operations in the Federal Multi-Site Evaluation of Foster Youth Programs

Purpose: In examining the effectiveness of social service programs, it is important to understand not only whether the program had the intended impact on its participants, but also how the program was implemented, how it operates, and what challenges to implementation and operations exist. This paper will present findings from the process study component of the Multi-Site Evaluation of Foster Youth Programs in two Los Angeles County programs: the Early Start to Emancipation Preparation (ESTEP) – Tutoring program and the Life Skills Training (LST) program. The process study sought to answer several research questions: (1) does service implementation follow the programs' logic models; (2) how are services implemented; (3) what are barriers to implementation; and (4) what are the challenges to program operations? Methods: To answer these questions, the process study used qualitative methods. Data collection primarily focused on in-depth fieldwork, conducted at two points in time, which included semistructured interviews and focus groups with program staff, child welfare agency staff, other stakeholders, and youth, as well as observations of program staff serving youth. In total, process study staff interviewed over 200 different respondents. The process study also collected quantitative data on program take-up and dosage. Results and Conclusions: The ESTEP and LST programs are different programs which target different ages of foster youth. ESTEP-Tutoring provides one-on-one tutoring in basic reading and math skills to foster youth ages 14 to 15. LST provides 30 hours of life skills classes on a community college campus to foster youth ages 16 and older. In general, the ESTEP and LST programs adhere to their logic models. However, as with any social service program, they have experienced some challenges to implementation and service provision, including working with a mobile foster youth population and adapting the program to meet each individual youth's needs. At the same time, these programs offer unique advantages, such as providing transportation to and from the LST classes and offering tutoring in the youth's home. This paper draws on the process study component of the evaluation of these two programs to offer insights and implications for other service systems about engaging youth clients, particularly those in foster care. For instance, the LST program has specific staff members whose primary responsibility is to recruit youth for the program, which includes telephone and in-person outreach. It also provides specific information about how these programs operate and handle service challenges. For example, the ESTEP-Tutoring program offers tutoring in the home so that transportation (for the youth) does not prevent youth from participating in the program. Finally, the paper highlights the need for rigorous quantitative and qualitative evaluation of social service programs in order to fully understand program impacts and operations.