Interrater Agreement of the Group Care Quality Standards and the Group Care Quality Standards Assessment

Background and Purpose:

The Family First Prevention Services Act (Family First) emphasizes the importance of children growing up in families and when foster care is needed, being placed in the least restrictive, most family-like setting appropriate to their special needs. As a result, Family First limits the use of group care to Qualified Residential Treatment Programs, which are required to demonstrate family engagement, use a trauma-informed treatment model, and to be licensed and accredited. However, existing measures of quality of group care are limited and mostly not quantifiable.

This study is part of a research project to develop the Group Care Quality Standards and the Group Care Quality Standards Assessment (GCQSA), a quantifiable measure of quality of group care. Developed by researchers and practitioners in Florida in 2014, the GCQSA contains 8 domains, each of which has 4-20 items. The domains include: 1) Assessment, Admission, and Service Planning, 2) Positive, Safe Living Environment, 3) Monitor and Report Problems, 4) Family, Culture, and Spirituality, 5) Professional and Competent Staff, 6) Program Elements, 7) Education, Skills, and Positive Outcomes, and 8) Pre-Discharge/Post Discharge Processes. Different forms of the GCQSA were designed for raters of different titles: youth, group care providers (i.e., directors and director care workers), lead contract agency staff (e.g., case managers, contract managers, placement coordinators), and state licensing specialists.

In this study, we evaluated interrater agreement (IRA) of GCQSA. IRA is the absolute consensus in rating scores from multiple raters on the same targets. High IRA justifies aggregation of scores from multiple raters.

Methods:

We used state-wide data collected in 2018-2019 from 189 group homes (GHs) licensed by the Florida State Department of Children and Families. We used the r_WG(J) index as the IRA measure, which defines agreement in terms of the proportional reduction in error variance. We used the r_WG(J value of 0.50, the cut point of moderate agreement to indicate acceptable IRA, since most GHs were rated by only 2 raters of the same title, which is a small number of raters and can attenuate the value of r_WG(J).

Results:

The results showed that for the forms used by direct care workers, director/supervisors, and lead agency staff, Domains 2-7 showed acceptable IRAs, indicated by that at least 60% GHs have moderate-high IRA with the values of r_WG(J) ranging from 0.50 and 1. For the youth form, Domains 2 and 6 showed acceptable IRAs, indicated by that at least 60% GHs have moderate-high IRA. For the other domains on the youth form, at least 50% GHs have moderate-high IRA.

Conclusions and Implications:

Among the four forms of GCQSA, most domains showed acceptable IRA. Therefore, these findings lend support for aggregating ratings from multiple raters of the same title to provide a composite score on quality of residential care. Future research needs to examine the inter-rater agreements between raters of different titles. Our finding further supports that GCQSA has the potential to be used as a reliable measure to measure quality of group care, in response to Family First.