Assessing Psychometric Properties of a Nonprofit Human Service Organization Capacity Survey

Background and Purpose:

Capacity-building is a strategy to help nonprofit human service organizations (NPHSOs) strengthen performance in achieving their missions (Light, 2004), yet no standardized measures exist for NPHSO capacity (Wing, 2004). The purpose of this study was to validate a model for measuring NPHSO capacity to assess the effectiveness of NPHSO capacity-building interventions and more accurately assess administrative factors that affect the implementation of evidence-based practices and programs in NPHSOs (Fixsen, Blase, Naoom, & Wallace, 2009).

Methods:

Data for this study were from the baseline survey of the Compassion Capital Fund (CCF) Demonstration Program Outcome Study (ICPSR 29482). A representative of each NPHSO (N = 1,221) completed a survey at baseline and 15 months after receiving capacity-building assistance. The survey included 54 items related to capacity in organizational, program, revenue, and leadership development, and community engagement. Three conceptual models were assessed: 1) a model with 54 items and 5 domains specified by prior CCF research (Minzner et al., 2010; 2014); 2) a model with 54 items and 4 domains used by the Marguerite Casey Foundation (2012) based on prior conceptualizations (Connolly & York, 2003; McKinsey & Company, 2001); and 3) a model with 19 items and 4 domains using CCF level-of-focus (i.e., self-assessed progress) items only. Confirmatory factor analysis (CFA) was used as a theory-driven technique (Schreiber et al., 2006) to assess fit among the three models using mean- and variance-adjusted weighted least squares (WLSMV). Group invariance was examined by tenure.

Results:

Most (58%) NPHSOs in the sample had annual revenues less than $100,000, compared to 40% of nonprofits in the U.S. (Pettijohn, 2013), and most (63%) had been serving the community for less than 10 years. The CCF and Marguerite Casey Foundation models failed to fit the data well based on multiple fit indices, even after poor performing items were dropped. However, the CCF level-of-focus model with 19 items measuring NPHSOs’ self-assessed progress in developing capacities in four domains, fit the data well (χ² = 288, df = 140, p<.001; RMSEA = .042, [90% CI] = .035, .048; CFI = .959; TLI = .950) after correlating the measurement errors of conceptually similar items and verifying fit with a validation sample. Group invariance tests were statistically non-significant, indicating fit applicable to both younger and older NPHSOs.  

Conclusions and Implications:

Because the CCF and Marguerite Casey Foundation models failed to fit the data well, it may be NPHSO capacity is not well conceptualized by either of these models and/or that it is difficult to accurately measure organizational performance with such a large number (54) survey items. Another possibility – and a key limitation of the CCF data – is that a single NPHSO representative provided survey responses. Still, the third model, comprised of level-of-focus items only, can be used by practitioners to identify and assess areas for building NPHSO capacity in resource development, board development, management capacity, and program development.