Mismeasuring Impact: How Randomized Controlled Trials Threaten the Nonprofit Sector

Background: What makes human service nonprofits effective? There are many potential answers to this question, but increasingly, funders of nonprofits, especially governments, seek quantitative measurements of program effectiveness as proof that social interventions “work.” Nonprofits are increasingly asked to align with the evidence-based policy movement which is pushing governments to restrict social policy funding to those programs with an established evidence base, for which the randomized controlled trial (RCT) is seen as the “gold standard.” This presentation highlights the main findings from a recently published book that explores how RCTs came to be seen as the most legitimate form of evidence, how RCTs are really carried out inside nonprofits, and where there is a mismatch between the RCT method and nonprofit goals. We go beyond existing critiques about RCTs to address unintended consequences for equity, sustainability, and organizational responsiveness and innovation. The key research question is: Are randomized controlled trials a good method for understanding the performance of human service organizations?

Methods: Our findings are based on a field analysis that included archival work as well as 54 interviews with professionals involved in the RCT ecosystem: professional evaluators with experience conducting RCTs nationally (N=17), foundation program officers from foundations with assets sufficient to fund RCTs and established funding in the human services (N=16), and nonprofit managers, both those that had participated in an RCT and those that chose not to (N=21).

Results: We identify five specific problems with using RCTs as high-stakes assessments of social interventions:

1. The “False Certainty” Problem: The RCT methodology is a bad match for the field conditions in most nonprofits, frequently underpowered, and poorly implemented and controlled.
2. The “Programs Need Organizations” Problem: RCTs assess programs, but programs are embedded in organizations. While RCTs may tell us which program are effective, organizations can experience damaged finances and reputations, threatening program sustainability.
3. The “Communities Need Organizations” Problem: RCTs threaten the community-level benefits provided by nonprofit organizations by rewarding organizations exclusively on improving individual outcomes that can be evaluated with an RCT
4. The “Rich Get Richer” Problem: RCTs primarily advantage already well-resourced organizations whose way of working is easily adapted to RCT demands. Smaller organizations, or those serving specific cultural contexts or serving minority populations, may not be able to be labeled ‘effective’ or ‘evidence-based’ due to the limitations of the method (not the programs).
5. The “Agility” Problem: RCTs hinder responsiveness and innovation because of their long time horizon and lack of course correction opportunities.

Implications: Instead of increasing reliance on RCTs, our findings suggest we should invest in evaluation methods that are tailored to the organizational-community interface, responsive to the concerns of beneficiaries, and allow for iterative and ongoing learning. These evaluation methods have the added advantage of alignment with social work values regarding the importance of human relationships and challenging inequalities. Using RCTs to make up-or-down decisions about whether specific programs work will rarely lead to the kind of nonprofit improvement that other forms of learning and evaluation can provide.