Abstract: A Case Study of Algorithmic Decision-Making in Child Welfare Agencies: Frontline Practitioner Experiences and Perspectives (Society for Social Work and Research 27th Annual Conference - Social Work Science and Complex Problems: Battling Inequities + Building Solutions)

All in-person and virtual presentations are in Mountain Standard Time Zone (MST).

SSWR 2023 Poster Gallery: as a registered in-person and virtual attendee, you have access to the virtual Poster Gallery which includes only the posters that elected to present virtually. The rest of the posters are presented in-person in the Poster/Exhibit Hall located in Phoenix A/B, 3rd floor. The access to the Poster Gallery will be available via the virtual conference platform the week of January 9. You will receive an email with instructions how to access the virtual conference platform.

A Case Study of Algorithmic Decision-Making in Child Welfare Agencies: Frontline Practitioner Experiences and Perspectives

Saturday, January 14, 2023
Ahwatukee A, 2nd Level (Sheraton Phoenix Downtown)
* noted as presenting author
Amanda Farley, Implementation Associate, University of North Carolina at Chapel Hill, NC
Audrey Loper, MPH, Implementation Specialist, University of North Carolina at Chapel Hill
Daniel Gibbs, MSW, JD, Doctoral Candidate, University of North Carolina at Chapel Hill, Chapel Hill, NC
Allison Metz, PhD, Professor and Director of Implementation Practice, University of North Carolina at Chapel Hill, NC
Background and Purpose: In 2020, child welfare agencies received and responded to 3.9 million reports of maltreatment involving 7.1 million children. The decision to screen and respond to child protection referrals can be highly intuitive and requires a complex weighing of risk and protective factors and an extensive review of administrative records regarding families’ histories. The outcomes of these decisions have been shown to demonstrate significant inaccuracy and unreliability, prompting some agencies to turn to algorithmic decision-making (ADM) tools to increase the quality and fairness of the screening process. Little research has addressed the ways in which such tools are implemented and integrated into workers’ existing processes of clinical judgment. Accordingly, this qualitative case study of two child welfare agencies utilized participant observation, semi-structured interviews, focus groups, and document review to (1) describe the manner in which ADM tools are used in real-world practice conditions and (2) examine practitioner perspectives regarding whether and under what conditions these tools are useful in improving their decision-making.

Methods: Twenty-six interviews were conducted with caseworkers, supervisors, and leaders across two county agencies following a review of twenty-one key practice documents and seven observations of tool use across multiple cases. Member-checking of initial findings was then conducted through eleven focus group sessions with thirty-nine workers, supervisors, leaders, and tool developers and evaluators. Data collection focused on obtaining a clear description of the practices and values associated with tool use and eliciting participants’ views about the usefulness of the tool in their work, how its recommendations were weighed alongside other case factors, contextual influences on tool usefulness, and recommendations for further development. All interviews and focus groups were recorded and transcribed for thematic analysis.

Results: Data analysis and triangulation provided a robust depiction of the implementation and practice activities involved in using ADM tools for screening decisions. Participants described a process in which the output of the tool was accessed by supervisors in a team decision-making setting, its predictions were compared with extensive discussion of referral content and prior history, and a group decision was made based on all available information that may or may not align with the prediction. Staff—particularly supervisors—viewed the tool as a compelling source of additional information and accountability, but practitioners disclosed significant barriers to usefulness including misalignment with referral content and statutory requirements, technical difficulties, limited understanding and explainability, and difficulty using the tool in group decision-making and research study contexts.

Conclusions and Implications: The use of ADM tools continues to be a promising approach for improving decisions in human services contexts. However, the challenges described by frontline staff regarding the tool’s ability to be integrated into clinical and group decision-making practices suggest that further innovation and research may be needed regarding the design and implementation of tools that are responsive to their human users’ needs. Computer scientists, child welfare researchers, administrators, and policymakers, should think critically about ways that practice-grounded realities and the mechanics of intuitive judgments can be assessed and incorporated into future ADM intervention efforts.