Sensitivity and Specificity of the Screening Decision in a Canadian Child Protection Agency

Background:

Deciding which referrals to investigate is a critical decision for child protection agencies. In Canada, the province of Ontario has the highest incidence of investigations, without a correspondingly higher rate of verified maltreatment, suggesting that the screening threshold is lower in this jurisdiction. There is pressure on children’s aid societies to investigate fewer cases, both due to limited resources and an acknowledgement that investigations that do not confirm the need for child welfare service may damage relationships in well-functioning or already vulnerable families. To explore this, a large urban CAS conducted a sensitivity/specificity analysis to ascertain the degree to which screeners accurately identified families in need of child protection services.

 

Methods:

Three fiscal years of data (N=8,501) were extracted from the agency’s information system, based on an entry cohort, i.e., all cases referred during this time period, using the first referral in the fiscal year as the “index” referral). Cases screened in at the index referral and verified through the investigation were considered true positives (TP) while cases screened in and not verified within 12 months were categorized as false positives (FP). All cases screened out at the index referral were tracked forward for 12 months to determine if they were the subject of a subsequent referral resulting in an investigation and verified protection concerns during this time period. Cases screened out at the index referral that did not experience a verified re-referral within 12 months were considered true negatives (TN). Those cases screened out at the index referral and which were the subject of a verified re-referral within 12 months were regarded as false negatives (FN).

Sensitivity was calculated as: Se = _ TP___

                                                         TP + FN

Specificity was calculated as: Sp = _ TN___

                                                         TN + FP

Additional data were extracted to explore factors associated with “false positives”, including the primary reason for referral, the referral source, and the worker making the screening decision.

 

Results:

Results indicate that while the sensitivity of the screening decision is high (Se=96%), the specificity is quite low (Sp=39%), with screeners escalating 61% of cases in which child protections concerns were not present to investigation. Analyses suggest that factors associated with false positives include referrals from school personnel, and/or allegations of physical abuse. A few individual workers were more likely to make decisions that resulted in false positives; however, the overall variance at the worker level was minimal.

 

Conclusion:

Reducing unnecessary intervention in families that do not require child protection services is a goal for the Ontario child welfare sector. In a time of scarce resources and public sentiment that child protection services can be overly intrusive, there is pressure on CASs to screen in fewer cases. This analysis provides guidance for staff charged with the screening decision to think critically about the cases that they screen. It has also provided the foundation for a series of strategies developed within the agency to work differently with other systems (e.g., education) to support the needs of vulnerable children and families.