Critical Analysis of Organizational Risk Threshold

Background and Purpose: The Child Welfare System in Ontario Canada has been under significant public criticism over the past few years. A number of child death inquests have been highly publicized, while the cost of providing child welfare services has continued to increase. This has contributed to a crisis in confidence – do we do to much or too little? When risk thresholds are set too low it can over-burden a child welfare system resulting in significant strain on limited resources and subject families unnecessarily to an intrusive investigation. However, a risk threshold that is too high can result in harm to children, including death. To explore one Ontario Child Welfare organizations’ risk threshold a sensitivity/specificity analysis was conducted. This analysis was then used to facilitate critical reflection on current disposition decision-making related to the which families are investigated.

Methods: Based on methods developed by Mansell et al. (2011), data were drawn from Family and Children’s Services of the Waterloo Region’s client database to examine the first referral or ‘spell’ per family between 2012 and 2017 (N = 14,163). Each case was tracked for 12 months to investigate the sensitivity and specificity of the referral disposition decision. Four groups were derived from this analysis: True Positive (screen in and verified within 12 months); True Negative (screen out and not verified within 12 months); False Positive (Screen in and not verified); False Negative (screen out and verified within 12 months). Bivariate analyses were used between the four groups and the decision to investigate. For False Positives only, additional analyses were conducted to explore correlates of these outcomes. File reviews were also completed using random selection per year.

Results: While the sensitivity of the screening decision is high (Se=93%), the specificity is low (Sp=43%). Of cases investigated and substantiated only 36% of cases were transferred for ongoing services. Factors associated with false positives include referrals from schools and allegations of physical abuse. 25% of all false positive cases involved inappropriate discipline, 20% related to a caregiver risks, and 14% involved exposure to IPV.

Conclusions and Implications: The analysis allowed the organization to rethink the response to families who present as lower risk or who do not present with imminent safety concerns. More than 8,000 families over the five-year period were the subject of an intrusive investigation without any services or supports being offered. The fear and anxiety regarding false negative decisions appeared to be driving the organization's risk threshold down, yet only 2.7% of cases fell within this false negative grouping. The results of this analysis were used to facilitate discussions at an organizational level about whether the risk threshold and service responses fit the presenting issues faced by families referred to the organization. The study recommended that the service response provided to the profile of families within the false positive group should look substantively different from an intrusive child welfare investigation. As a result, the percentage of cases disposed to investigation dropped from 80% to 60% over the 12-month period.