Methods: Twenty-six interviews were conducted with caseworkers, supervisors, and leaders across two county agencies following a review of twenty-one key practice documents and seven observations of tool use across multiple cases. Member-checking of initial findings was then conducted through eleven focus group sessions with thirty-nine workers, supervisors, leaders, and tool developers and evaluators. Data collection focused on obtaining a clear description of the practices and values associated with tool use and eliciting participants’ views about the usefulness of the tool in their work, how its recommendations were weighed alongside other case factors, contextual influences on tool usefulness, and recommendations for further development. All interviews and focus groups were recorded and transcribed for thematic analysis.
Results: Data analysis and triangulation provided a robust depiction of the implementation and practice activities involved in using ADM tools for screening decisions. Participants described a process in which the output of the tool was accessed by supervisors in a team decision-making setting, its predictions were compared with extensive discussion of referral content and prior history, and a group decision was made based on all available information that may or may not align with the prediction. Staff—particularly supervisors—viewed the tool as a compelling source of additional information and accountability, but practitioners disclosed significant barriers to usefulness including misalignment with referral content and statutory requirements, technical difficulties, limited understanding and explainability, and difficulty using the tool in group decision-making and research study contexts.
Conclusions and Implications: The use of ADM tools continues to be a promising approach for improving decisions in human services contexts. However, the challenges described by frontline staff regarding the tool’s ability to be integrated into clinical and group decision-making practices suggest that further innovation and research may be needed regarding the design and implementation of tools that are responsive to their human users’ needs. Computer scientists, child welfare researchers, administrators, and policymakers, should think critically about ways that practice-grounded realities and the mechanics of intuitive judgments can be assessed and incorporated into future ADM intervention efforts.