Developing and Implementing a Predictive Risk Model to Support Child Protection Supervisors: Results from Los Angeles County Risk Stratification Pilot

Background and Purpose. With the growth of “predictive analytics” and “predictive risk modeling” in other fields, child protection agencies are increasingly exploring how the increased availability of data and algorithms can support improved decision-making, less biased determinations of risk, and a more intentional delivery of services. In the current study, we detail the development and piloting of a risk stratification model to assist supervisors in Los Angeles (LA) County better manage information.

Methods. We extracted all referrals received by LA between 2016-2017. This period allowed a two-year follow-up window during which we could observe outcomes prior to any COVID-19 pandemic lock-downs. We identified all individuals named on a given referral and coded them into different roles (e.g., victim, perpetrator). A set of more than 300 features were generated to describe different characteristics of individuals and features associated with the referral overall (e.g., time and day, number of children). Each victim child-referral was labeled to document the outcome used for training: whether the child was placed in foster care within 24-months of the referral date. We split child victim-referral observations into a training set (75%) and a test (or “hold-out”) set (25%). The data were split to ensure that a victim child was exclusively assigned either in the training or the test set. This process left us with a total of 341,428 victim child-referral observations representing 171,313 unique referrals.

The choice of the modeling method came down to ease of deployment. Earlier state level modeling had experimented with Random Forest and established a benchmark performance using state data. A baseline Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression model was estimated on the final referral cohort with the outcome labeled. We trained the LASSO model using weighted classification to account for the low prevalence of the outcome, using 10-fold cross-validation in 3 repetitions. The hyper-parameters were chosen to maximize the Area Under the Receiver-Operating Characteristic Curve (AUC-ROC): 83.0 (95% CI: 82.6-83.5).

Three applications were implemented to draw upon data from the model. These included: (1) a flag provided to supervisors overseeing emergency response investigations, alerting them to new investigations the model classified as “complex-risk”; (2) an investigation history report that summarizes information that can be time consuming to assemble through the existing case management system; and (3) a racial-equity report and feedback loop that will be used by LA DCFS to examine low-complexity referrals that were screened in for investigation and involved Black children.

Results. Supervisors reported several enhanced practices extending from the model. These included more frequent meetings with the assigned investigator, focused reviews of history, teaming, and earlier engagement with community partners. Some staff reported challenges in the adoption of new technologies; others expressed concern that these investigations required more time than permitted.

Conclusions and Implications. Data from the first 6-months of the pilot did not indicate any unintended consequences. Preliminary findings from the racial equity reviews suggest at least three opportunities for diverting more investigations to community pathways.