Society for Social Work and Research

Sixteenth Annual Conference Research That Makes A Difference: Advancing Practice and Shaping Public Policy
11-15 January 2012 I Grand Hyatt Washington I Washington, DC

16321 Using Existing Scales to Assess Child Welfare Workforce Characteristics: How Scales Fail

Saturday, January 14, 2012: 11:00 AM
Independence D (Grand Hyatt Washington)
* noted as presenting author
Inna Altschul, PhD, Assistant Professor, University of Denver, Denver, CO
Cathryn C. Potter, Professor, Associate Provost for Research, Executive Director, Butler Institute for Families, University of Denver, Denver, CO
Jennifer Middleton, PhD, LCSW, Assistant Professor, University of Maine, Orono, ME
Background and Purpose: When developing comprehensive surveys, researchers often rely on previously developed, and, preferably, widely-used scales to assure meaningful results. Our team did just that in selecting the Professional Quality of Life (PQL; Stamm, 2005) and Job Satisfaction (JS; Spector, 1985) Scales to assess child-welfare workforce characteristics. Each scale has been widely used and appeared to have sound psychometric properties. In this paper, we present psychometric results for these two scales using data from 1,192 child-welfare workers, and discuss ways in which they failed to perform adequately in measuring intended constructs. We offer recommendations to other researchers on how to select measures to avoid similar pitfalls.

Methods: Scales were administered online as part of a larger child-welfare organizational health assessment in four states. Participation was voluntary with 1,192 surveys completed. The sample was split into two random halves to conduct exploratory (EFA; n=586) and confirmatory factor analyses (CFA; n=606) in Mplus 6.1. Because these scales had been previously vetted we began with CFA to fit models as described in the literature. If CFA models did not adequately fit the data, we conducted EFA to determine factors structures that would fit the data, and confirmed the new structures with CFA.

Results: Psychometric examinations of PQL and JS scales showed that they do not adequately represent the constructs they were intended to measure, and cannot be used in structural equation models as intended. The resulting scale based on PQL items bears little resemblance to the original scale; of 30 items, four remained representing Work Satisfaction (Fit Statistics: χ2(2)=1.366, p=0.51, CFI=1.000, RMSEA=0.071, SRMR=0.006, alpha=.852). With regard to Job Satisfaction, 32 of 36 original items were retained and six of nine subscales confirmed; however, three subscales did not fit the data as intended. More importantly, the scale cannot be combined into one higher-order construct representing job satisfaction. Scales failed to assess constructs adequately in at least four ways: at the individual item-level with vague or “double-barreled” statements; at the subscale-level with items representing latent constructs not overlapping enough in substance or there not being enough items to represent each subscale construct; and at the broader construct level with subscales not fitting together to form one larger construct as described in the literature.

Conclusions and Implications: In this case, use of existing and widely-used scales did not result in psychometrically sound measures. We provide a typology of ways in which scales fail. We use the PQL and JS scales as examples; however, it is important to note that many other existing scales in our assessment tool failed similarly, and none performed as intended. Our recommendations to other researchers interested in building psychometrically sound assessment and survey tools are to: look for measures that have been examined in an EFA/CFA framework, rather than simply had reliability confirmed via Chronbach's alpha; examine all item wording for applicability in your setting; examine groups of items for evidence of an underlying latent construct; and expect to conduct extensive psychometric analyses prior to structural equation modeling.

<< Previous Abstract | Next Abstract