A multi-method approach for assessing fidelity to Family Connections

BACKGROUND AND PURPOSE. The degree to which programs balance fidelity to core components of a program with local adaptations without sacrificing program efficacy remains a scientific question that necessitates the documentation of modifications made to programs while examining program outcomes. One focus of the FC Replication cross-site evaluation was to examine the quality of implementation of FC, including adherence to and modification of the core components of FC. This paper presents the process for developing the methods for the cross-site fidelity assessment, the fidelity rating scheme, and selected results about the extent to which FC replication projects implemented core components of FC with fidelity. METHODS. The development of the cross-site fidelity assessment began with operationalizing the FC fidelity criteria. Multiple data sources for each criterion were identified and a rating scheme was developed. This resulted in a multi-method assessment of adherence to the cross-site fidelity criteria/essential elements of FC. Methods included abstracting archival data (i.e., implementation plans, site-specific intervention manuals, semi-annual report narratives, and data in a fidelity database), coding case records, conducting semi-structured interviews with program staff and community partners, and facilitating focus groups. The central focus of this paper is to report on the degree to which program staff employed core elements of this social work intervention. Replication site case records were examined to determine the extent to which program staff adhered to fidelity criteria associated with conducting the initial comprehensive family assessment and developing service plans that included goals that were specific, measurable, achievable, realistic, and time limited (SMART). The cross-site evaluation randomly selected up to five case records per replication site frontline staff member for each site per six-month period between October 1, 2004 and September 30, 2007 (i.e., 6 six-month time windows) to allow for the examination of implementation fidelity between staff, sites, and over time. A total of 467 records across eight sites were double-coded for adherence to the fidelity criteria. RESULTS. Results from the cross-site fidelity assessment suggest that there was variation in the degree to which the FC Replication sites implemented the core components of FC with fidelity. For example, data available from the cross-site case record review indicate that FC project staff adhered to key fidelity criteria such as conducting comprehensive family assessments and incorporating standardized measures into service planning and provision at varying levels. FC replication sites improved adherence to these components over time and as a result of training and technical assistance from the FC mentor agency. CONCLUSIONS AND IMPLICATIONS. The findings from the case record review suggest that over time, replication sites increased adherence to fidelity criteria associated with conducting a comprehensive family assessment and tailoring case plans to the findings from the family assessment. Participants will be encouraged to discuss the complexities of measuring core dimensions of social work practice. The meaning of these findings for social work practice with families at risk of child maltreatment and for future research will also be discussed.