Saturday, 15 January 2005 - 2:00 PM

This presentation is part of: Workforce Issues in Social Work

Measuring Organizational Learning In Human Services: Development and Validation of the Organizational Learning Capacity Assessment

Purpose: The purpose of this paper is to describe the psychometric properties of the Organizational Learning Capacity Assessment (OLCA). The OLCA measures the extent to which a human services organization operates with the structural and cultural components necessary to promote organizational learning and effective change management strategies. Previous research on successful interventions in human services demonstrates that many interventions that have proven scientific merit are not adopted by the host organization. Intervention success is not only the result of offering high-quality, evidence-based practice models but also tied to the ability of an organization to accept and adopt new information and intervention technologies as part of their organizational practices. This measure can help organizations assess their learning capacity.

Methods: This is the third of four validation studies designed to test and optimize the psychometric properties of the OLCA using a bi-national sample from Israel and the US. In the initial pilot of the instrument, a 67-item questionnaire including thirteen dimensions of learning was constructed based on pilot qualitative research. Using exploratory factor analysis, expert reviews, and external raters, this instrument was trimmed to a 40 item questionnaire that included ten dimensions of learning. For the present study, the 40 item instrument was given to a bi-national sample of after-school program staff members as part of an 18-month longitudinal study designed to implement organizational learning as a management practice in the programs. Staffs in six sites (2 in the US and 4 in Israel) were given the instrument at three time points: Baseline, 6 months, and 18 months. Alpha reliability and factor analysis was conducted on responses to establish reliability of the subscales. Item analysis using t-tests were conducted to assess predictive validity of each item for the whole sample and for respondents in each country.

Results: Based on the longitudinal analysis, the instrument was reduced to 18 items. Nine items describe structural components of learning in organizations, such as time set aside for meetings, availability of materials to support learning, agency dissemination of new practices, etc. These items and the subscale successfully discriminate between the treatment and control groups at Time 1 and Time 2. Nine additional items correspond with cultural components of learning and assess agency beliefs in support for learning, attitudes toward tolerance for errors, beliefs in leadership support for learning, etc. These items and the subscale also successfully discriminate between the treatment and control groups at Time 1 and Time 2. The retained items demonstrated adequately high factor loadings (.57 or above) in each subscale. Alpha reliability of the two subscales is high (.82 to .84).

Implications: Quantitative measures that assess the extent to which human services organizations are learning and capable of integrating best-practices can be a critical tool to help ensure successful implementation of new interventions. Rigorous, longitudinal psychometric testing of such measures is key to Social Work’s ability to promote evidence-based practice models. A confirmatory analysis of the findings of this study using the18 item index with a larger bi-national sample is currently being conducted.

See more of Workforce Issues in Social Work
See more of Oral and Poster

See more of Celebrating a Decade of SSWR (January 13 - 16, 2005)