Process Evaluation of Dialectical Behavior Therapy Dissemination: Knowledge Retention and Recursive Training Effects

Background: Wider dissemination of evidence-based cognitive therapies, such as dialectical behavior therapy (DBT), will require development of new models for implementation that fit the needs and constraints of community-based providers. DBT can be challenging and resource-intensive to implement with fidelity to the model, requiring substantial initial training, ongoing consultation, and reinforcement by the program milieu in which it is delivered. Residential treatment settings are frequently overlooked as sites of implementation for DBT despite the severity of client symptoms, and elevated rates of staff turnover and burnout. This presentation will outline a cost-effective training and process evaluation model, along with the impact of the training on provider DBT-specific knowledge, staff turnover rates, and staff perceptions of training impact on their practice.

 Methods: Five residential programs serving individuals with severe mental illness, chemical dependency, and/or forensic involvement in the greater Philadelphia area participated in the training initiative from 2013-2016. Staff DBT-specific knowledge was measured pre- and post-training through administration of an 18-question assessment of principles and skills associated with the four DBT modules: emotional regulation, distress tolerance, interpersonal skills, and mindfulness (n=58). Pre- and post-test results were analyzed. Demographic variables were correlated with post-training knowledge assessment scores. Annual staff turnover rates prior to and following DBT training were calculated for each participating program individually and collectively. Individual, semi-structured qualitative interviews were conducted with staff from each program participating in the training (n=8). Initial, independent, open coding was conducted by the two interviewers. Categorization was completed and the conceptual model was refined through discussion. A data display of the model was created using CMaps software. Conceptual model and themes were reviewed for accuracy with agency staff. 

Results: Scores on the DBT Knowledge Assessment Quiz ranged from 50-100%, with an average score of 83.2% on the 18-item quiz (M= 14.97; SD= 2.3), compared with a pre-test mean score of M=12.9; SD= 2.1. Post-training scores represent a statistically significant increase (t=4.14; p=.001). Although educational level did have a significant effect on quiz total final scores (f= 3.19; p= 0.02), over three-fourths of the sample (77.6%) scored 83.2% or higher on the quiz. Age and length of tenure were not significantly correlated with total quiz scores (f=2.23; p=0.14; f=0.16; p=0.90). Program staff turnover rates did not vary significantly pre- and post-training in four of five programs despite showing an overall trend in reduction. A significant decrease in staff turnover was seen when all five programs were analyzed together (p=0.02). Qualitative analysis of staff interviews suggests recursive training effects consistent with DBT model. Themes emerged related to impact of the training on three primary areas: effectiveness of practice with clients, application of DBT skills in staff personal lives, and positive effects on program culture.

Conclusion: Results suggest that it is feasible to train residential staff in the principles of DBT, but attention is required to uneven knowledge retention. The recursive training model may also benefit staff personally and professionally. Further study of reduction in staff turnover is needed. Implications for successful dissemination of DBT will be discussed.