Lessons Learned to Date from the Qic-WD Cross-Site Evaluation

Background and Purpose: The QIC-WD cross-site evaluation seeks to understand turnover in child welfare systems. To accomplish that goal, the cross-site evaluation ensures that similar outcome variables are captured in all eight sites so that a meta-analysis of results can be conducted. These include survey as well as Human Resources (HR) and Child Welfare Administrative outcomes housed at each site. The survey outcomes include the same measures of job satisfaction, organizational commitment, intentions to leave the agency, and organizational culture and climate. The HR outcomes include turnover. The Child Welfare outcomes include such outcomes as reunification rates, re-entry rates, types of out of home placements (e.g. level of restriction), and length of time in out of home placements. One of the predictor variables for child welfare outcomes is turnover. Thus, the evaluation team has examined various ways to consistently calculate turnover for that purpose.

In assessing turnover as an outcome variable, a common formula for calculation is number of staff who leave the agency in a month/number of staff employed at the beginning of that month plus the number of staff hired during the month. These monthly data can then rolled up to calculate quarterly and annual turnover rates (stock and flow). Another possible way to calculate turnover is to examine turnover by cohorts of employees (cohorts). Cohorts can be defined as employees entering the agency at roughly the same time (e.g., month, quarter or year) or as employees present at the beginning of the QIC-WD study. There are also several ways to calculate turnover longitudinally. In order to test the efficacy of different turnover calculation methods, several site datasets were used.

Methods: Data from two sites were utilized, Oklahoma and Virginia. Survival analysis was utilized across a two-year period for cohorts. Oklahoma HR data from 2016-2018 were utilized and subjected to survival analyses. In Virginia staffing patterns in the 18 participating counties in the QIC-WD study are gathered monthly from March 2018. The sample from March 2018 to March 2020 was used to estimate turnover using cohort based event history analyses (survival analysis), and alternatively, based on overall quarterly rates of turnover (stock and flow). These two methods were used to formulate quarterly time series of turnover estimates and were associated with workforce size and other child welfare indicators.

Results: We found in comparing methods (cohorts vs. stock and flow) there were differences in the association between turnover and staffing level over time with event history being more highly correlated with trends in staffing levels.

Conclusions and Implications: Results from this analysis reveal that differences in methods for estimating turnover are important in achieving a clearer understanding of how turnover operates at a systemic level within child welfare agencies. Developing and testing methods for estimating turnover in site and cross-site analysis is key to understanding how changes in turnover and other system level workforce characteristics might or not be related to child welfare outcomes.