Development of a Multidimensional Approach to Assessing Organizational Health

Background:  Continued efforts by human services agencies to improve workplace climate and workforce retention requires accurate assessment of key aspects of organizational functioning. However, few such measures have been developed and tested specifically for use with human services agencies. The Comprehensive Organizational Health Assessment (COHA) survey battery was designed to fill the need for a set of validated tools that assess the complex constellation of organizational factors that influence worker performance and turnover rates, ultimately influencing service delivery and outcomes for children and families. The COHA assessment strategy includes quantitative (survey data), qualitative (focus groups and interviews), and contextual (document and data review) components. The goal of this study is to describe the rigorous strategies employed to develop and refine a battery of measures that have demonstrated ability to quantitatively gauge organizational health.

Method: The COHA has undergone an iterative development process to adequately capture multiple dimensions of organizational health such as: burnout and intent to stay (individual-level), supervision and professional support (unit/office-level), leadership and workplace climate (organizational-level), and community resources and perceptions of child welfare work (community-level). Administered to a large sample of child welfare workers, initial approaches included calculating reliabilities and subjecting each measure to factor analysis to determine the number of and alignment with hypothesized factors or subscales. Based on results, items were adjusted, new measures were added as needed, and the process was repeated again with another large sample.

Results: Data presented here are from 2,976 staff from child welfare agencies in three states who were administered the COHA survey that included approximately 300 items across 24 measures. We first established the individual measures’ reliability (Cronbach’s alphas ranged from .72 to .98), then inter-correlated all measures in the battery to determine whether they capture distinct constructs. Highly correlated scales (r >.70) were analyzed together using EFA, while the others were tested individually with Rasch analysis (item response theory; IRT) for unidimensionality and to determine the range of difficulty of items and whether they functioned differently for subgroups. These analyses resulted in removal of two measures, and merging of others.

Implications: When measuring organizational functioning of human services agencies, it is important to consider which aspects of the climate to measures and the appropriate unit of analysis for that construct. Although finding standardized measures is an important first step, it is equally important to check for content validity (the domains of greatest interest) and face validity (items capture desired constructs). If standardized tools do not exist and tool development is warranted, the new tool should be rigorously tested for internal consistency, the existence of subscales or factors, differential functioning of scales or items with subgroups of the population, and convergent or predictive validity. Results of this study indicated that the COHA battery reliably measures a wide range of important organizational factors and is a source of valid measurement with child welfare staff, making it ideal for assessing the health of the child welfare workforce and organizational factors that influence readiness for change interventions and practice efficacy.