![[ Visit Client Website ]](images/banner.gif)
Implementation fidelity is an important component of intervention research as it assesses adherence to an intervention protocol and manual, improving reliability of results (Borelli et al. 2005). Fidelity is especially important in multi-site or large-scale studies, where multiple clinicians may be delivering the intervention. Without assessment of fidelity, it is difficult to conclude that an intervention was effective and delivered consistently across sites. AFFIRM is an eight-session, evidence-informed, manualized group cognitive-behavioral (CBT) intervention designed to improve coping and reduce depression among sexual and gender minority youth (SGMY). AFFIRM is currently being tested in a stepped-wedge trial with community implementation, with numerous sites and facilitators engaged in its delivery. In an effort to highlight social work science for social change, this presentation will explore the iterative development of the AFFIRM Fidelity Checklist, and the results of an initial fidelity assessment over a three-year implementation period.
Methods
The AFFIRM Fidelity Checklist was iteratively developed by community/academic stakeholders using the processes described by Feely et al. (2018): (1) defining the purpose/scope of fidelity assessment, (2) identifying essential components of the fidelity monitoring system, (3) developing the fidelity tool, (4) monitoring fidelity, and (5) using the fidelity ratings in analyses. Four raters (three MSW-level, one BSW-level) received 16 hours of coding training and were randomly assigned to rate audio-recorded (2-hour) sessions (N=151), totalling 300 hours of coding. Inter-rater reliability estimates were generated using 35% of sessions (N=53), randomly selected for double coding. The inter-rater agreement was examined by calculating the proportion of observed agreement between raters, and Cohen’s Kappa Statistic. To calculate Cohen’s Kappa statistic and the observed agreement, the score was transformed into 5 categories—Poor Fidelity to High Fidelity (Landis & Koch, 1977). Each AFFIRM Facilitator (primarily social workers, psychologists, and community workers) received a 2-day (14-hour) training on the intervention, and monthly supervision.
Results
The AFFIRM Fidelity Checklist included 8 key fidelity indicators with a total possible of 100 and space for score justification. The indicators included items pertaining to implementing AFFIRM as intended (e.g., adherence to manualized protocol, CBT model, group work skills). A total of 151 sessions were coded. The mean fidelity score is 83.91 (SD=11.33), which suggests high fidelity and aligns with extant implementation fidelity literature (Hepner et al., 2011). Cohen’s Kappa statistic resulted in an alpha of 0.6, with the proportion of observed agreement between raters resulting in 81%, suggesting substantial fidelity agreement.
Implications
This study evaluated group facilitators’ fidelity to affirmative group CBT (AFFIRM) for SGMY. This study has demonstrated that diverse, trained facilitators can consistently implement an affirmative group CBT intervention for SGMY. Study results suggest that facilitators highly adhered to AFFIRM intervention protocol, with good treatment fidelity and inter-rater reliability. A study limitation is the student status of the coders; experienced clinicians may derive more nuanced results. Training diverse community-based facilitators, particularly social workers, to deliver affirmative CBT is a promising approach to increasing access to effective mental health treatment. Implications for social work research, education, and practice with SGMY will be explored.