A Case Study of Algorithmic Decision-Making in Child Welfare Agencies: Frontline Practitioner Experiences and Perspectives

Background and Purpose: In 2020, child welfare agencies received and responded to 3.9 million reports of maltreatment involving 7.1 million children. The decision to screen and respond to child protection referrals can be highly intuitive and requires a complex weighing of risk and protective factors and an extensive review of administrative records regarding families’ histories. The outcomes of these decisions have been shown to demonstrate significant inaccuracy and unreliability, prompting some agencies to turn to algorithmic decision-making (ADM) tools to increase the quality and fairness of the screening process. Little research has addressed the ways in which such tools are implemented and integrated into workers’ existing processes of clinical judgment. Accordingly, this qualitative case study of two child welfare agencies utilized participant observation, semi-structured interviews, focus groups, and document review to (1) describe the manner in which ADM tools are used in real-world practice conditions and (2) examine practitioner perspectives regarding whether and under what conditions these tools are useful in improving their decision-making.

Methods: Twenty-six interviews were conducted with caseworkers, supervisors, and leaders across two county agencies following a review of twenty-one key practice documents and seven observations of tool use across multiple cases. Member-checking of initial findings was then conducted through eleven focus group sessions with thirty-nine workers, supervisors, leaders, and tool developers and evaluators. Data collection focused on obtaining a clear description of the practices and values associated with tool use and eliciting participants’ views about the usefulness of the tool in their work, how its recommendations were weighed alongside other case factors, contextual influences on tool usefulness, and recommendations for further development. All interviews and focus groups were recorded and transcribed for thematic analysis.

Results: Data analysis and triangulation provided a robust depiction of the implementation and practice activities involved in using ADM tools for screening decisions. Participants described a process in which the output of the tool was accessed by supervisors in a team decision-making setting, its predictions were compared with extensive discussion of referral content and prior history, and a group decision was made based on all available information that may or may not align with the prediction. Staff—particularly supervisors—viewed the tool as a compelling source of additional information and accountability, but practitioners disclosed significant barriers to usefulness including misalignment with referral content and statutory requirements, technical difficulties, limited understanding and explainability, and difficulty using the tool in group decision-making and research study contexts.

Conclusions and Implications: The use of ADM tools continues to be a promising approach for improving decisions in human services contexts. However, the challenges described by frontline staff regarding the tool’s ability to be integrated into clinical and group decision-making practices suggest that further innovation and research may be needed regarding the design and implementation of tools that are responsive to their human users’ needs. Computer scientists, child welfare researchers, administrators, and policymakers, should think critically about ways that practice-grounded realities and the mechanics of intuitive judgments can be assessed and incorporated into future ADM intervention efforts.