Developing a Contextually-Based Model for Program Evaluation Practice

Measuring program effectiveness has been a cornerstone of evaluation practice. The emphasis on evidence-based practice has historically focused on the “program” as the unit of study; excluding important contributions to effectiveness made by the context in which programs operate (e.g., staff characteristics, organizational structures and policies, norms, etc.). Within this, outcomes have generally been accounted for through fidelity of implementation. Rarely do current evaluation practices include the ways that habits, routines, values, customs, common meanings and traditions are expressed in the language and behaviors of practice (the cultural context of practice) and its impact on service outcomes. As a result, the outcomes to which the agency is held accountable (outcomes of accountability) may differ from the outcomes that are expected and desired by clients (outcomes of value). The current approach to the evaluation of program effectiveness has been cited as fueling the practice-research gap which exists within service delivery agencies and which promotes resistance to the use of empirically tested practices. Recent research has begun to acknowledge the influence of organizational factors on various aspects of service delivery including adoption and implementation of evidence-based programs and outcomes, although client systems have not yet been included in this line of research. This roundtable session will frame a discussion about organizational functioning in today's post-modern society by presenting a new model of evaluation practice that is currently being tested in three distinct organizations:  1) a Family Resource Center, 2) an undergraduate program within a School of Social Work; and 3) a Public Guardianship agency. This model expands evaluation practice beyond the program as the unit of study to a more contextually-based practice that recognizes both the outcomes of accountability and outcomes of value for each stakeholder group (administrators, practitioners, and clients). To begin the roundtable, the first presenter will provide an overview of this new model of evaluation practice. A second presenter will discuss current literature about organizational functioning and the gap that this model seeks to address. This section will emphasize the neglect of client constituents in current models which assess multiple levels of organizational factors as they influence service provision. The new evaluation model presented at this roundtable posits that program effectiveness is best served when there is alignment of outcomes of value and accountability throughout all the various levels of service delivery. The next four presenters will discuss methodological strengths and challenges in applying this model in real practice settings. Our goal is to stimulate a conversation about evaluation practice as it pertains to organizations that provide the context of the majority of social service delivery—including child welfare, mental health, and health care.