Building Partnerships Between Evaluators and Child Welfare Agencies through Continuous Quality Improvement and Instrument Development

Background:  Thirty states have active Title IV-E Waivers that require evaluation.  Evaluations use data provided by the Child Welfare Agency (CWA) and data collected by the Evaluators.  A trusted partnership is an important part of a formative evaluation approach.  For this demonstration, our evaluation team used a Continuous Quality Improvement (CQI) process in which the Evaluators were included, to collaborate on instrument development.

Methods:  Our team developed an annual online survey in 2013 to assess the perceptions of caseworkers in a CWA in a Midwestern state.  The first and second rounds of data collection included questions about workers’ most recently opened and closed cases along with their perception of the availability and use of service array for their clients.  For the first round of data collection, communication between the Evaluators and the CWA consisted of emails.  In Round 2, there was a one-hour meeting between the CWA and Evaluators before the survey launched.  Evaluators then presented the results of the first two rounds to Executive and Regional Management.  With feedback from the presentation, and the inclusion of the Evaluation team in the CQI process, the  Round 3 development consisted of three meetings, totaling five hours, multiple emails and testing by the CWA, and the addition of close to 80 new questions.

Results:  For the first and second iterations, the instrument consisted of 33 and 38 questions respectively, which took case managers an average of 39 minutes in Round 1 and 35 minutes in Round 2 to complete.  The overall response rate was 75% in Round 1 and 67% in Round 2 with 91% of surveys useable for analyses in Round 1 and 83% for Round 2.  With the increased collaboration time and the CWA’s investment in the instrument, the third iteration remarkably improved with an 85% response rate, 95% of which were usable for analysis.  Round 3 took 56 minutes to complete on average.  The additional questions provided more meaningful and timely data that the CWA was able to use in service planning.

Discussion:  As the CWA began to realize the power of data and developed more trust in the evaluation team, instrument development was stronger and resulted in not only increased data, but more meaningful data.  Through collaborative efforts, the Evaluators were able to gather more data on a wider array of topics that were important to the CWA and the evaluation of the IV-E Waiver.  These data are able to be used with other administrative data in a more cohesive and powerful manner which can then be used in the field to affect practice.  This exercise also led to greater reliance in the Evaluators and the development of more instruments, such as community stakeholder instruments and CQI readiness instruments.