Testing for Differential Item Functioning, and Estimating Its Magnitude, Using Item Response Theory

Background and purpose: Numerous Likert-type measures are used in social work research and practice. It is critical that the validity of scores from these measures be invariant across the diverse populations who participate in social work research and for whom social workers provide services. A lack of validity invariance would indicate scores on these measures represent different constructs and need to be interpreted differently for persons from the different populations. This lack of validity invariance may lead to erroneous clinical and research inferences. A methodology for identifying differential validity of items is the differential item functioning (DIF) study. When items function differently for different populations, it is referred to as DIF. DIF can propagate to the total score on a scale producing differential test functioning (DTF), meaning the total score represents different constructs for different populations. Item Response Theory (IRT) is a useful approach for conceptualizing and identifying DIF and DTF. This workshop will provide participants with a basic knowledge of: (a) DIF and DTF through the lens of IRT concepts, (b) how IRT can be used to identify DIF and DTF in Likert-type scales using the program IRTPRO, and (c) how to use an online Excel program to estimate effect sizes for DIF and DTF.

Methods: First basic IRT concepts of the item response function and the item characteristic curve will be introduced and used to define two forms of DIF. The IRT concepts of the item and test information curves will be used to illustrate how DIF may propagate to the total scale score to produce DTF, thereby posing a serious threat to validity. Recently developed effect sizes for DIF and DTF will be described, and participants will be shown how to use the online Excel program to obtain estimates of these effect sizes. Participants will be walked step-by-step through a DIF analysis of Likert-type items from the Suicidal Thinking Subscale on Hudson's Multi-Problem Screening Inventory (MPSI) using IRTPRO. Participants will also be walked step-by-step through use of the online Excel program to obtain effect size estimates for DIF and DTF. These IRT methods will be related to confirmatory factor analysis methods for identifying DIF.

Results: Participants will learn how to conduct, and interpret the results of, confirmatory DIF and DTF analyses. Participants will also learn how to follow-up the identification of items exhibiting DIF with subsequent research to identify whether bias exists in the items showing DIF; to identify sources of the bias; and what can be done to eliminate the bias.

Conclusions and implications: Participants can expect to leave this workshop with a basic understanding of fundamental IRT concepts and the IRT conceptualizations of DIF and DTF; beginning skills in the use of IRT-PRO for conducting DIF and DTF studies; the use of the Excel program for estimating DIF and DTF effect sizes; and the ability to use the results from DIF and DTF analyses for developing measures for use in social work research and practice that are unbiased and culturally and population sensitive.