The Implications of Randomized Trials As a Means to Organizational Legitimacy

Background & Purpose: Randomized Control Trials (RCTs) have been viewed as the gold standard for intervention research and program evaluation since the 1980s. In attempts by funders and researchers to identify and support programs proven to be effective, the use of RCTs have proliferated across human service organizations (HSOs). RCTs as the mode of rigor in research and program evaluation is now tightly linked with efforts to maintain or attract outside funding among HSOs. However, we identify some unintended consequences: as HSOs increasingly partner with researchers and foundations to conduct RCTs, these efforts may bring hidden costs in data collection, front-line worker practice and buy-in, and client experiences. Therefore, the proliferation of RCTs in HSOs can result in unexpected implementation challenges for social workers within HSOs. Our research questions are: How do human service organization managers and front-line workers understand the burdens and benefits of RCTs? What leads to these varied perspectives?

Methods: In this paper we use qualitative data from an implementation study of an RCT that administered intensive case management services to the treatment group to reduce poverty and improve work at a large non-profit in the southwest United States. We draw from two waves of in-depth interviews and field observations with non-profit managers and front-line workers. We developed research questions based on existing literature that suggests research can place burden on workers and clients. Our protocol included questions on front-line staff perspectives on administrative burden, client engagement, and participating in an RCT. We coded a priori themes particular to our research protocol. We then re-coded for emergent themes. A major emergent theme across administrators and front-line staff was the importance of the RCT model to organizational legitimacy among donors and the research community, including demonstrating positive effects of the program on client outcomes. Based on our deductive analysis of the data, we illuminate previously unexplored organizational implications that accompany the use of RCTs.

Results: We find that HSO managers often view RCTs as a way to build or maintain organizational legitimacy among key stakeholders—foundations, researchers, and private donors. However, managers often fail to anticipate the costs of RCTs experienced by both workers and clients. These costs include difficulty in maintaining client engagement, worker resistance and discretion, staff turnover, and decreased organizational trust from front-line workers.

Conclusions & Implications: We find that viewing RCTs as legitimacy-seeking endeavors provides a framework for understanding how the burden of investing in an RCT may, in fact, challenge or complicate the benefits. While this mode of research has increasingly dominated funding and research, this study helps challenge researchers to consider the implications of RCTs within an organizational context. These insights are especially relevant to researchers and funders who often engage in and endorse RCTs. Finally, this framework aids social workers who work within funding agencies or foundations, manage HSOs, or provide front-line services to better prepare for the potential impact of RCT evaluation on organizational practices.