Evidence Clearinghouses As Policy Tools: How Inequitable Treatment of Evidence Eroded Tool Legitimacy Among Implementing Organizations

Background: Spurred by evidence-based policy reforms, evidence clearinghouses have emerged across policy domains in the United States. Framed as innovations that bridge knowledge-practice gaps, evidence clearinghouses identify effective practices via systematic review processes that prioritize experimental evidence and disseminate information about “what works.” Although advocates suggest that this model ensures a more effective social safety net with limited public resources, others question whether only approving programs assessed in controlled environments discounts community-based programming and marginalizes organizations’ and communities’ tacit knowledge, considered especially important when providing services to minoritized populations. Passed in 2018, the Family First Prevention Services Act (FFPSA) is the first child welfare legislation to both establish an evidence clearinghouse and tie its use to federal reimbursements. As evidence clearinghouses grow in use, understanding how they are interpreted and used on the ground is critical to effective policy.

Methods: Our longitudinal qualitative study followed diverse Minnesota child welfare stakeholders for two years as they developed the state’s FFPSA Prevention Services Plan. Data include 48 semi-structured interviews with key stakeholders involved in plan development and 100+ hours of participant observation across 13 workgroups. Transcripts and field notes were coded using inductive and deductive approaches and extensively analyzed during bi-weekly team meetings.

Results: Implementing organizations identified equity concerns about four Clearinghouse features that undercut its legitimacy as a useful policy tool. First, respondents asserted that an evidence review and rating system requiring experimental evidence to demonstrate practice effectiveness and receive reimbursements a) privileged organizations with greater resources to mount expensive practice evaluations and b) positioned practices evaluated only with white populations as universally effective. Second, unsure who the Clearinghouse administrators tasked with reviewing and rating practices were, respondents serving minoritized communities hypothesized that administrators were “east coast White researchers” who might not recognize, or – perhaps more importantly – be the right people to recognize, the effectiveness of community-based and Indigenous practices. Third, limited transparency about how to get practices rated expediently undercut faith in the Clearinghouse’s ability to respond to organizations’ urgent implementation needs. Fourth, practices with lower Clearinghouse ratings required ongoing monitoring efforts that were cost-prohibitive for less-resourced organizations (typically serving minoritized communities). Ultimately, minimizing Clearinghouse-associated implementation requirements drove decision-making, leaving Minnesota with a Prevention Plan comprised of well-rated, reimbursable Clearinghouse practices yet unresponsive to organizations’ visions of equity.

Implications: Although the Clearinghouse model is designed specifically to advance the legitimacy of some programs over others, our respondents questioned the legitimacy of the tool as a whole. Implementing organizations’ perceptions of the Clearinghouse’s inequitable treatment of knowledge and ignoring of implementation conditions eroded its legitimacy as a tool responsive to community needs. If policymakers continue to utilize clearinghouses as policy tools, such tools must be redesigned to be inclusive of multiple forms of evidence and accompanied by sufficient resources so that local organizations can address community needs and meet implementation requirements. Organizational scholarship must consider how well-intentioned policy tools may incentivize organizations to implement practices that run counter to organizations’, communities’, and social work values and to what effect.