Discretion in Fidelity Practice: How Frontline Workers Navigate Artificial Intelligence While Meeting Federal Policy Compliance Demands

Background

Federal policymakers increasingly suggest that successful governance hinges upon local actors’ uptake of evidence-based practices (EBPs) and the tools that aid their implementation. Accordingly, evidence-based and techno-scientific governance reforms have thrust new accountability demands on local providers already trying their best to manage multiple accountability relationships with funders, partners, and clients given limited time, resources, and staff. This paper explores how frontline workers manage multiple accountabilities when the evidence-based movement and algorithmic governance converge in daily practice via a policy case: the Family First Prevention Services Act (FFPSA). In 2018, Congress passed FFPSA, for the first time linking federal reimbursement to child welfare implementers’ use of and demonstrated fidelity to EBPs. While FFPSA stipulates that implementers must complete fidelity monitoring for each EBP, how they do it is their choice. Some child welfare organizations are responding by adopting artificial intelligence (AI) fidelity monitoring tools. Subsequently, federal compliance requirements are passed down to frontline workers, now tasked with engaging AI tools with clients in practice.

Methods

This paper draws data from an ongoing project – a qualitative case study of child welfare organizations using AI fidelity monitoring tools to meet compliance demands when implementing FFPSA. Data include 120+ observation hours and 40 interviews with staff (25 frontline; 15 leadership) at a child welfare nonprofit in a small rural state contracted to implement one of the state’s EBPs: Motivational Interviewing (MI). Data include frontline workers’ thoughts about and experiences engaging an AI tool that records MI sessions with clients and generates reports about their MI fidelity. Theory-based deductive codes and data-driven inductive codes informed a qualitative codebook; the author completed iterative thematic analyses across data sources.

Results

Preliminary analyses suggest that organizational leadership remain unsure what data the federal government will require and by when to secure reimbursements. Leadership perceive AI fidelity monitoring tools as one strategy for covering as many data bases as possible, at scale, when the rules of the game are unclear. Frontline workers’ perceptions of the tool are mixed. While some view it as an opportunity to develop MI skills, most experience the AI tool as psychologically unsafe for them and clients and the newest iteration of check-the-box accountability to meet compliance demands. Frontline workers’ decisions about when and how to use the tool vary: some use it with “chatty” clients, believing it will capture and accurately evaluate robust MI data; others use it when session content is “light” to protect both clients’ personal experiences from being recorded and the therapeutic alliance; still others use it when a recording deadline draws near, regardless of client or content.

Conclusions

While AI tools hold promise for worker skill development, most frontline workers in this study experienced AI fidelity monitoring tools as another form of check-the-box accountability prioritizing institutional pressures over practice needs. Absent clear guidance from the federal government, and subsequently organizational leadership, about compliance data purpose, requirements, and consequences, frontline staff perceived AI tools as burdensome, resulting in wide discretion in fidelity practice, ultimately undercutting federal policy aims.