Early Results from the California Learning Collaborative Evaluation

Background and Purpose: In 2006 nine California county mental health organizations agreed to participate in a group learning collaborative for quality improvement led by the California Institute for Mental Health and the Center for Healthcare Strategies. The goal of the learning collaborative was to improve quality in community mental health by adapting a QI model that has been successfully used in primary medical care settings, Best Clinical and Administrative Practices (BCAP) (Ha, 2005). The learning collaborative required that program directors and other administrative staff attend a series of workshops and group meetings over a two year period, during which time they would develop, pilot, and evaluate a new mental health project based on BCAP principles. The purpose of this evaluation of the learning collaborative was to:

1.Understand factors involved in the effectiveness of the learning collaborative

2.Understand the factors related to implementation, dissemination, and sustainability of the BCAP model and related processes

3.Understand the impact of the learning collaborative and BCAP on the counties' QI processes, project implementation capacity, and overall organizational culture

Methods: The unit of analysis was the county, and the goal was to develop county case studies that could “tell a story” of the county's participation in the learning collaborative process and be used as a basis for cross-case analysis. Three sources of data were used:

1.Non-participant observation, in which researchers openly disclosed their role, primarily observed, and minimally participated in learning collaborative meetings and technical assistance calls.

2.Interviews with staff directly and indirectly involved in the learning collaborative projects (n=30).

3.Content analysis of learning collaborative documents, such as worksheets, powerpoint presentations, and handouts

Results: There were important differences in the nature and scope of the counties' projects. The extent of BCAP adaptation may have been related to the scope of the county project—the more “diffuse” the project, the greater the adaptation. Measurement activities and associated learning seemed highly congruent with county objectives, but, counties experienced difficulty gathering the data required for BCAP. Participants reported high levels of satisfaction with the group learning environment despite the lack of central “common indicators” across counties, enormous competing resource demands, and a decline in attendance for those in top leadership positions. Participants reported successes in obtaining buy-in to new processes from clinic staff. As a result of the overall satisfaction with the learning collaborative experience, many counties are actively planning to replicate projects to other sites, but they have not made any firm commitments to continue using BCAP.

Conclusion and Implications: The learning collaborative provided a useful vehicle for quality improvement in county mental health agencies. However, there may be a poor fit between current BCAP tools and the decision-making and resource environment of community mental health. Most counties had to develop new data management processes, layered on existing systems, and this encumbered BCAP adaptation. The BCAP model is promising, but further research is needed to refine BCAP for community mental health systems.