"There's Always Sort of a Consciousness Around the Results, Right?": A Case Study of a Large Nonprofit's RCT Implementation As Policy Entrepreneurship

Background

Consensus among researchers, donors, administrators, and policy practitioners has established randomized control trials (RCTs) as a “gold standard” for evidence in policy making, expanding use from fields like medicine, public health, and ecology to human services and poverty reduction programs. RCTs have also evolved from evaluating large-scale global development interventions to evaluating hyper-local nonprofit domestic programs where mission, values, and service often take precedence. For those nonprofit actors attempting to influence policy, RCTs can offer advantage in the marketplace of policy solutions. How might nonprofit-based RCTs be used for means other than evaluation? Further, how might these uses shape RCT implementation?

Methods

To explore these dynamics, we present an organizational case study that consists of ethnographic (300 hours of agency observations over 2.5 years) and interview data (administrators and caseworkers, N=42) at two points in time. The case is a program that was initiated by a large organization in the southwest United States, whom we call “the Agency.” The Agency collaborated with researchers on a multi-faceted social service based RCT (2015-2018), which we call the “All In” program. The RCT and research partnership were initiated by the Agency to test and prove the program’s efficacy. The intervention included services delivered by case teams, a lead case worker (LCW) who was primarily responsible for client engagement services and weekly contact, and a case assistant (CA) who was primarily responsible for bureaucratic demands and data entry. The All In RCT also included discretionary emergency financial assistance (EFA) and 12 partner agencies to deliver concrete support and material resources outside of the scope of the Agency’s services. Outcome measures for clients in the treatment group included securing a living-wage job (based on county median household income), three months of savings, and no debt. The control group received the usual Agency services if desired.

Findings

We find that institutional entrepreneurs at the Agency prioritized generating a legitimated path from RCT-tested program to policy entrepreneurship. If the program was proven to “work”, it could be added to state and federal clearinghouses, be piloted as a replacement for entitlement programs, or be funded via government contracts. To this end, we present evidence that program administrators, acting as institutional entrepreneurs, were keenly aware of the role RCTs play in shaping state and federal policy. In this way, some nonprofit leaders used their own agency to employ an RCT with the end goal of policy entrepreneurship.

Discussion

This study contributes to a growing literature on RCT implementation in nonprofits and the currency of rigorous evidence in policy. However, underlying ambitions for generating policy change can generate tensions for both RCT implementation fidelity and frontline staff’s intentions to provide equitable and responsive program implementation that “meets clients where they are”. This case study aids us in understanding the ways in which the valuation of evidence, itself, is reproduced within HSNPs not simply via state mandates or contacts but instead via an HSNP’s desires to influence policy, or shift power and resources in an organization’s external environment.