Accountability in Intervention Research: Developing a Fidelity Checklist for a Randomized Controlled Trial of a Mental Health Intervention in Prison

Background: A critical component of intervention research is developing fidelity tools to assess treatment adherence and to ensure equivalence of experience for individuals receiving an intervention. However, fidelity tools remain underutilized in intervention research and few resources exist to guide their development in real-world service settings. This study presents the development of a fidelity checklist for a complex, two-part intervention delivered to people with serious mental illness (SMI) in prison. The first part of the intervention is a manualized, CBT, group intervention. The second part is a Targeted Service Delivery Approach where facilitators use a flexible set of treatment strategies to address the neurocognitive and social impairments associated with SMI when delivering the intervention. This paper presents the development of the fidelity checklist for this intervention and methodological considerations shaped by the prison context at key decision points.

Methods: The study assembled a fidelity team to develop a fidelity checklist using the five-step process developed by Feely and Colleagues. These steps include: 1) defining the purpose and scope of the fidelity assessment used, 2) identifying essential components of the fidelity monitoring system, 3) developing the tool, 4) monitoring fidelity, and 5) using the fidelity ratings during analyses.

Results: The final checklist includes 26 items. Key methodological decisions were shaped by the study context, included determining the purpose and scope of the measure and selecting a method of administration. Since the central goal of fidelity checklists is to determine adherence to treatment models, the fidelity team created a checklist that assessed interventionists’ adherence to two different intervention components that varied significantly in terms of degree of structure. Developing a checklist that adequately captured both intervention elements was further complicated by the fact that key intervention elements included both content and process items. The team addressed this complexity by creating a checklist that assessed the presence/absence of key intervention elements using yes/no responses, rather than measuring the quality or quantity of treatment activities. This response format maximized efficiency and ease of completion of the checklist. This methodological decision was also important given that the treatment setting for this study created challenges around the methods of administration for the checklist; prisons ban all recording devices, making direct observation the only method allowed. However, using outside observers could create reactivity among participants due to the small treatment space available in prison and the distrust and suspicion that pervade correctional settings. Therefore, the only method of administration available in this study was for both interventionists to complete the checklist.

Implications: Fidelity measures the extent to which an intervention is delivered as intended. Failure to ensure treatment fidelity impedes our ability to link study outcomes to the intervention. However, the development of fidelity measures is resource-intensive and shaped by the context in which it will be used. The development of fidelity requires balancing the rigor of the tools and the needs of service settings. This study presents information for researchers to use when developing methodologically rigorous fidelity measures that are applicable to real world settings.