Policy Making and Randomized Designs in Child Welfare: A Case Study of the Multi-site Evaluation of Foster Youth Services

Purpose: This paper will describe the development of a major random assignment evaluation in child welfare and discuss the policy implications of the impact findings from the first two sites in the study. The Foster Care Independence Act (FCIA) gave states more funding and greater flexibility in supporting youth making the transition to independent living. The FCIA requires an unique evaluation of programs. Few child welfare programs have been subjected to rigorous evaluation. The U.S. Department of Health and Human Services contracted with the Urban Institute, Chapin Hall Center for Children, and the National Opinion Research Center at the University of Chicago to conduct this evaluation. This presentation will discuss the development of the evaluation and the implications for policy of the impact and process findings from two sites. Method: The research team first conducted an evaluability assessment to identify programs that could be rigorously evaluated and to develop evaluation designs that would meet the requirements of the authorizing legislation. The evaluation team established criteria for selecting sites for the evaluability assessment. DHHS selected programs to be evaluated. Programs were required to exhibit the following: sufficient numbers of youth, excess demand for services, program stability, intensive services, well-developed theories of intervention, consistent implementation, available data, and support of decision makers for random assignment. Thirty-two states and the District of Columbia were contacted and 87 different independent living programs were examined. Site visits were conducted with the 23 programs that seemed most promising. The legislative mandate required a random assignment design. We will discuss the implications of this mandate on the search for programs, as well as the challenges of conducting a random assignment design in child welfare settings and the ways the methods have affected the policy impact of the study. Results: Four programs were selected for inclusion in the evaluation. (For two sites, follow up has been completed and impact results will be discussed in a separate paper on this panel.) These four programs are not representative of independent living services available to youth in the United States. Rather, they represent a range of different interventions that IL programs use. As a result, the findings from the Multi-Site Evaluation cannot be generalized to all independent living programs. Results will be released during the summer of 2008. Implications: The findings from the first two sites in the study will be released to the public in the coming months. We will assess how these findings are integrated into evolving policy on youth aging out of foster care. The evaluation results raise important questions for the child welfare policy and practice communities. It also raises questions about the appropriate role of randomized trials in the development of knowledge about social service programs.