Are You Ready to Randomize? Evaluators and Program Providers Discuss Participating in Rigorous Evaluation

Establishing evidence for program effectiveness is one of the most pressing needs in social work research. Aside from the moral imperative to provide effective assistance to people in need, the field is faced with increasingly urgent demands to demonstrate that investments in social services will yield positive returns, either in the form of improved outcomes for service recipients, or, better yet, in actual savings to governmental agencies and/or society as a whole. Establishing this evidence is not, however, an easy task.

This roundtable brings together two important perspectives: providers and evaluators of programs. The evaluators, who assess and document the evidence, will share information on important factors that might indicate an agency is ready, willing, and able to participate in rigorous evaluation. The providers will discuss how to prepare for and participate in rigorous evaluation, as well as how to most effectively utilize evaluation results.

Representatives from the team working on the federally-mandated evaluation of Chafee-funded programs for youth who are aging out of foster care will join this roundtable discussion. The team, from the Urban Institute and Chapin Hall at the University of Chicago, is currently studying promising programs in several sites serving youth transitioning out of foster care. Through formative evaluation, the team is assessing program readiness for rigorous evaluation. During this roundtable, the team will discuss findings from 1) their review of the field of independent living programs and the current limited evidence on program effectiveness, and 2) critical lessons about ways that programs need to increase their capacity for rigorous evaluation and ultimately contribute to the knowledge base on effective program practices.

Two groups will represent the provider perspective: Youth Villages and First Place for Youth. Both providers have substantial experience with program evaluation, including studies conducted by independent evaluators using rigorous methodologies. First Place for Youth has been working toward an impact evaluation of their program for transition-age youth and has completed a formative evaluation and feasibility study as steps along this journey. Youth Villages’ YVLifeset program was the subject of one of the largest random assignment evaluations to date of services for youth aging out of foster care; the agency has completed an impact evaluation and is currently working with their evaluators on analysis of the study data that will guide program improvements in response to the impact evaluation, as well as preparing for a second random assignment evaluation of the program.

The discussion will be guided by audience interest, but will include stories and lessons learned from both the evaluators’ perspective and from the providers’ side. Participants will learn about the critical characteristics of programs that indicate readiness for rigorous evaluation and will hear about hard-won lessons in implementing evaluation in the complex environment of navigating stakeholder interests, political uncertainty, and unstable funding streams. Tips and tools will be offered to providers who are considering embarking on this journey; evaluators who are interested in engaging in this work may also benefit from hearing both perspectives on this often difficult, but ultimately important, endeavor.