Abstract: Developing and Implementing a Predictive Risk Model to Support Child Protection Supervisors: Results from Los Angeles County Risk Stratification Pilot (Society for Social Work and Research 27th Annual Conference - Social Work Science and Complex Problems: Battling Inequities + Building Solutions)

All in-person and virtual presentations are in Mountain Standard Time Zone (MST).

SSWR 2023 Poster Gallery: as a registered in-person and virtual attendee, you have access to the virtual Poster Gallery which includes only the posters that elected to present virtually. The rest of the posters are presented in-person in the Poster/Exhibit Hall located in Phoenix A/B, 3rd floor. The access to the Poster Gallery will be available via the virtual conference platform the week of January 9. You will receive an email with instructions how to access the virtual conference platform.

Developing and Implementing a Predictive Risk Model to Support Child Protection Supervisors: Results from Los Angeles County Risk Stratification Pilot

Friday, January 13, 2023
Encanto B, 2nd Level (Sheraton Phoenix Downtown)
* noted as presenting author
Emily Putnam-Hornstein, PhD, John A. Tate Distinguished Professor for Children in Need, University of North Carolina at Chapel Hill, Chapel Hill, NC
Rhema Vaithianathan, PhD, Professor, Auckland University of Technology, Auckland, New Zealand
Stephanie Cuccaro-Alamin, PhD, Research Associate, University of Southern California, Los Angeles, CA
Jacquelyn McCroskey, PhD, Professor, University of Southern California, Los Angeles, CA
Eunhye Ahn, PhD, Assistant Professor, Washington University in Saint Louis, MO
Claire McNellan, MPH, Doctoral Student, University of North Carolina at Chapel Hill, Chapel Hill, NC
Background and Purpose. With the growth of “predictive analytics” and “predictive risk modeling” in other fields, child protection agencies are increasingly exploring how the increased availability of data and algorithms can support improved decision-making, less biased determinations of risk, and a more intentional delivery of services. In the current study, we detail the development and piloting of a risk stratification model to assist supervisors in Los Angeles (LA) County better manage information.

Methods. We extracted all referrals received by LA between 2016-2017. This period allowed a two-year follow-up window during which we could observe outcomes prior to any COVID-19 pandemic lock-downs. We identified all individuals named on a given referral and coded them into different roles (e.g., victim, perpetrator). A set of more than 300 features were generated to describe different characteristics of individuals and features associated with the referral overall (e.g., time and day, number of children). Each victim child-referral was labeled to document the outcome used for training: whether the child was placed in foster care within 24-months of the referral date. We split child victim-referral observations into a training set (75%) and a test (or “hold-out”) set (25%). The data were split to ensure that a victim child was exclusively assigned either in the training or the test set. This process left us with a total of 341,428 victim child-referral observations representing 171,313 unique referrals.

The choice of the modeling method came down to ease of deployment. Earlier state level modeling had experimented with Random Forest and established a benchmark performance using state data. A baseline Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression model was estimated on the final referral cohort with the outcome labeled. We trained the LASSO model using weighted classification to account for the low prevalence of the outcome, using 10-fold cross-validation in 3 repetitions. The hyper-parameters were chosen to maximize the Area Under the Receiver-Operating Characteristic Curve (AUC-ROC): 83.0 (95% CI: 82.6-83.5).

Three applications were implemented to draw upon data from the model. These included: (1) a flag provided to supervisors overseeing emergency response investigations, alerting them to new investigations the model classified as “complex-risk”; (2) an investigation history report that summarizes information that can be time consuming to assemble through the existing case management system; and (3) a racial-equity report and feedback loop that will be used by LA DCFS to examine low-complexity referrals that were screened in for investigation and involved Black children.

Results. Supervisors reported several enhanced practices extending from the model. These included more frequent meetings with the assigned investigator, focused reviews of history, teaming, and earlier engagement with community partners. Some staff reported challenges in the adoption of new technologies; others expressed concern that these investigations required more time than permitted.

Conclusions and Implications. Data from the first 6-months of the pilot did not indicate any unintended consequences. Preliminary findings from the racial equity reviews suggest at least three opportunities for diverting more investigations to community pathways.