Multidimensional Item Response Theory Analyses as an Alternative/Supplement to Confirmatory Factor Analyses in Evaluating Factor Structures and Measurement Invariance

Background and Purpose: The purpose of this study was to compare and contrast the use of Multidimensional Item response theory (MIRT) to Confirmatory Factor Analysis (CFA) in the assessment and evaluation of the factor structure and measurement invariance of the Participation in a Social Work Community of Practice (PSWCoP) Scale. Traditional CFA has long been the de facto choice when assessing dimensionality and measure invariance, even though it is frequently done in direct violation of the underlying assumptions. The use of IRT for evaluating the psychometric properties of psychosocial measures has steadily increased over the past 3 decades, but only recently has readily available software reached the capacity for conducting multidimensional IRT. The two primary research questions addressed in this study were: (1) Based on the results of IRT/MIRT analysis, does the measure of Participation in a Social Work Community of Practice (PSWCoP) exhibit a dimensional structure consistent with the proposed model?; (2)Does analysis of the PSWCoP data using CFA produce results consistent with those produced with IRT/MIRT analysis? Does IRT/MIRT analyses lead to the same conclusions regarding dimensional structure and psychometric properties of reliability, validity, and model fit as those based on CFA?

Method: A convenience sample of 521 students currently enrolled in a MSW program was recruited from 13 CSWE-accredited schools. Students completed an online version of the PSWCoP Scale. Data were analyzed using Conquest 2.0 MIRT analysis software and Lisrel 8.8 structural equation modeling software. MIRT and CFA analytic strategies were compared using real data from the Participation in a Social Work Community of Practice (PSWCoP) Scale, with an emphasis on identifying both complimentary and contradictory conclusions regarding the psychometric properties of the PSWCoP.

Results: Both MIRT results and CFA results yield the same factor structure for the PSWCoP, but the results differ substantially in the estimation of item parameters, scale/subscale reliability, and overall model fit.

Conclusions and Implications: The benefits of using IRT as an alternative/supplement to classical test theory (CTT) are well established, but only recently have software capabilities allowed for the estimation of more complex, multidimensional models using IRT. The primary benefit in using IRT instead of classical test theory (CTT) in scale evaluation is that IRT fixes the problem of item-person confounding in CTT. A major limitation of CTT-based analyses such as Confirmatory Factor Analysis (CFA) is that parameter estimates are always sample-dependent. A second benefit derived from using IRT instead of CTT is derivation of interval level data from non-interval level raw scores. The use of ordered response formats (i.e., Likert scales) is frequently accompanied by the false assumption that the data are measured at the interval level; that is, the progress across response categories is treated as if it were ordered and consistent instead of simply ordered. A third benefit of IRT over CTT analyses is the assessment of invariance of parameter estimates in sub-group analysis. Assessment of measurement invariance in CFA requires large and equivalent sub-group sample sizes, while no such sample size constrictions exist in IRT analysis of invariance.