Self-Assessment of Assertive Community Treatment Fidelity: A Pilot Study

Purpose:

Measurable standards are critical in service assessment, and historically, evaluation of ACT has relied on costly, time-consuming fidelity visits.  Beyond the several days of observation, interviews, and chart review of these visits, quality measures are infrequently considered in teams’ day-to-day practice.  The development and introduction of the self-assessment of fidelity for ACT teams aims, first, to familiarize New York State ACT teams with regular fidelity assessment in practice; and in doing so, to support the longer-term goal of administrative and policy-level consideration of this cost-effective practice. 

Methods:

As part of a webinar series, the Tool for Measurement of Assertive Community Treatment (TMACT), was presented to ACT teams, to illustrate the concept of fidelity assessment.  Following consultation and review of the relevant literature, the TMACT was adapted as a self-assessment.  Using Qualtrics survey software, an online, user-friendly assessment of 49 items was made accessible to ACT teams to measure their fidelity.  Alignment of all items with New York State ACT Standards of Care was reviewed, and a 3-item subscale assessing Clinical Transition Activities was added in consideration of the emerging focus on transition from ACT.  In August, 2015, notification with a direct link to the tool was distributed by email to New York State ACT teams (n=81). 

Results:

Since its distribution, the self-assessment has been completed by a small number of teams (n=4).  For each of these teams, individualized feedback reports have been developed. The reports include subscale score reviews, identified strengths and areas for improvement, and specific resources and recommendations related to demonstrated needs. A record of teams’ scores, as well as the feedback reports shared, is maintained.  We observed that half of the teams who conducted assessments received face-to-face consultation prior to the assessment submission.  The reported time required to complete the assessments was less than a week for all teams, and while the team leader was identified as the person completing the final assessment for all teams, half of the teams reported collaboration with other team members.  All recommended sources of information were considered by at least one of the teams.

 

Implications:

The introduction of Health Homes by the Department of Health directly impacts New York State Assertive Community Treatment (ACT) teams in several ways.  In the midst of healthcare system transformation, teams have been faced with the addition of reporting regulations, among other changes.  The documentation standards are intended to ensure continuity of care and communication between providers, while regularly assessing clients’ readiness for transition from ACT to a less intensive level of care.  We engage providers in discussion of obstacles to their implementation of recommended practices.  We have learned of confusion about the function of this fidelity tool, and must continue to clarify and distinguish the purpose of the TMACT self-assessment from the evaluation process of fidelity site visits.  We recognize that re-consideration of ACT as a transition-focused service is also challenging for many teams.   Review of teams’ experiences with the measures continues to inform the support provided to teams by the ACT Institute.