WITHDRAWN: Using Machine Learning to Train and Test a Predictive Risk Model for Implementation

Background and Purpose. Each year, Los Angeles (LA) County receives and screens more than 80,000 referrals alleging abuse or neglect for nearly 160,000 unique children. Correctly assessing immediate concerns that may compromise a child’s physical safety is critical. Yet also critical, and a far more challenging, is assessing a child’s risk of future harm. With the growth of “predictive analytics” and “predictive risk modeling” (PRM) in other fields, child protection agencies are increasingly exploring how the increased availability of data and algorithms can support improved decision-making and resource allocation. In the current analysis, we detail the development and implementation of a risk stratification model to support supervisors during investigations in LA.

Methodology. Roughly 300 predictors coded and tested during the modeling process were derived from raw data tables originating in administrative records maintained as part of California's child welfare data system. Predictors were used as modeling inputs to train the risk stratification model. The outcome used for training purposes was an out-of-home placement within 2-years of the referral receive date. The model was trained using a LASSO regression method. A 75% sample of referrals that were screened-in for investigation during this two year period as training set, while the remaining 25% of screened-in referrals were held out during the training process and used for model testing purposes. Data from 2014-2016 were used to build the model; outcomes were tracked through 2019. To ensure there was no cross-over between our training and testing samples, we selected unique child-referral observations into each sample. This prevented two children on the same referral from being split so that one child was included in the training data and a second in the testing data.

Results. We evaluated accuracy of the model using the Area Under the Receiver-Operating Characteristic Curve (AUC-ROC) on the (25%) test set. We also looked at the True Positive Rate (TPR) and Positive Predictive Value (PPV) metrics on the selected model, after converting predicted probabilities on the test set into risk scores in the range 1-10, with each risk-score bin of the same size (same number of child-referrals). Findings indicate an AUC of 0.84. For children with a risk score of 10, 62% were placed in foster care within the next 2-years. Additional model validations using near-fatality and fatality data (i.e., outside of system indicators) validated model predictions.

Conclusions and Implications. Findings from the model reinforce it is possible to use administrative information available at the outset of a referral to identify a child's risk trajectory. This information can be used to guide the deployment of supervision and other resources to ensure the protection of children and family connections to services.