Interrater Reliability of Adverse Childhood Events: It Depends on Who You Ask

Background and Purpose: Adverse childhood experiences (ACEs) are associated with poor health and psychiatric illness across the life course. There has been a recent increase in the use of ACEs as predictors of health outcomes in both research and practice. However, limited research has explored who should be reporting ACEs for children. Existing research has shown experiences of family violence are reported differently by children and their caregiver. Similarly, children and parent reports on child health status and well-being do not always agree. This study will use a sample of children from the child welfare system to examine reporter agreement among children, caregivers, and caseworkers.

Methods: Data was drawn from the National Survey of Child and Adolescent Well-Being (NSCAW II), a longitudinal, nationally representative survey of children who have been the subject of a child welfare investigation. Information was collected from three sources (child, caregiver, and child welfare caseworker). The NSCAW survey does not collect ACE data from children until age 8. Therefore, we selected a sub-sample of participants aged 8-17 from Wave 1 data collection for a total of 1,887 participants. The following ACEs indicators had responses from children, caregivers, and caseworker: sexual victimization, physical abuse, emotional abuse, and home violence. Descriptive and bivariate analyses were completed to understand patterns of reporter’s responses to each of the ACEs selected. Using survey commands, analyses were adjusted for the cluster-based sampling design of NSCAW. Sampling weights were applied to provide nationally representative estimates to describe the sample. We calculated interrater agreement and reliability using the Kappa coefficient for each ACE indicator.

Results: Children, parents, and caseworkers differentially reported the number of ACEs the child experienced. Children reported experiencing three ACEs on average, while caregivers reported two and caseworkers reported an average of one. Differences between child, caregiver, and caseworker reports were often distinct on single items. For example, 85% of children reported experiencing violence in the home, while only 26% of caregivers and 38% of caseworkers reported this ACE. Informant agreement using kappa ranged between 14%-93% and varied by ACE indicator. Caregivers and children had high agreement reporting emotional abuse (74%) and caregivers and caseworkers had high agreement reporting child sexual abuse (93%). Overall, Kappa coefficients were extremely low with caseworker-caregiver agreement the highest at .29.

Conclusions: Screening for ACEs is important for prevention and targeted intervention strategies. However, as more researchers and practitioners utilize ACE screening to inform practice and policy, understanding the implications of who the reporter is will become critical. Our study identified children self-reported the highest number of ACEs. This finding could indicate parents and caseworkers may underreport ACEs for children. Given the lack of interrater reliability, we recommend asking multiple informants while carefully considering self report by children when assessing for ACEs.