Fidelity and Child Welfare System Reform: Creating Measures of Fidelity to Large-Scale Practice Changes

This roundtable will address measurement of fidelity in the context of broad system-level change, with exemplars from the field of child welfare practice. The authors will present information on measure development, collaboration with stakeholders, sustainability issues regarding fidelity data collection and interpretation, and intricacies of fidelity data measurement in the context of individual-level and organizational-level interventions.

Fidelity to an intervention is an essential component of implementation. Adherence to manualized treatment protocols is standard in intervention research, and measures for this type of fidelity review are well-established and validated. Although this is relatively common in individual level, evidence-supported interventions, the process for establishing and measuring fidelity is less clear in the context of large-scale systems change. Important fidelity-related considerations still apply to larger changes. We need to know how well we are implementing what we intend, requiring development of measures specific to large-scale systems change in child welfare. The authors will report on the measurement development process with state and local child welfare system partners.

The authors have developed fidelity measures in concert with public state child welfare system partners, as part of large-scale system change initiatives. Methods for assessing fidelity were designed with the intent to capitalize on and build the capacity of the system to monitor the degree to which core elements of the initiative were implemented as intended. These newly-developed fidelity measures were subject to pilot testing and refining, as well as inter-rater reliability testing. Fidelity reviews were ultimately integrated into continuous quality improvement systems. Throughout the process, authors sought to engage stakeholders, collaborate on measurement design, create meaningful and comprehensive items at the individual and organizational levels, promote sustainability through capacity building within the system, and feed back fidelity data to enhance the quality of practice. In this workshop, example strategies reflecting each of these elements will be shared.

During this roundtable, attendees will be encouraged to respond to questions such as the following:

1. What processes should be considered to operationalize core elements of organizational-level initiatives so that fidelity can be measured both simply and comprehensively?

2. What are the challenges in adopting a fidelity mindset among the child welfare workforce and leadership?

3. How can a set of practice principles be operationalized into a meaningful measure of fidelity?

4. What information should be assessed in all case records, versus records selected for review?

5. What elements of fidelity are best understood and measured at the individual client or family level? At the organizational level?

6. How can fidelity data be used to improve implementation efforts? Can fidelity data be misused?

7. How can fidelity measures impact the organization?

8. How can the presumed relation between improved fidelity and improved child and family outcomes best be assessed?

9. What can researchers do to promote sustainablility of fidelity measure use?