Chronically Inaccurate? Assessing the Reliability and Validity of the Service Prioritization Decision Assistance Tools for Persons Experiencing Homelessness

Background and Purpose:
Homelessness is a complex social crisis shaped by structural inequities and intersecting vulnerabilities. With limited housing assistance vouchers available, tools that accurately and equitably assess vulnerability among people experiencing homelessness (PEH) are critical. The Service Prioritization Decision Assistance Tools (SPDATs)—including the SPDAT, F-SPDAT, VI-SPDAT, and F-VI-SPDAT—are widely used to prioritize access to housing, yet concerns persist about their psychometric validity and fairness across diverse populations. This study presents one of the largest and most comprehensive evaluations of all four SPDAT variants to date, leveraging statewide data from Utah’s Homeless Management Information System (UHMIS) from 2014 to 2022.

Methods:
A retrospective analysis of 57,767 UHMIS records was conducted across three linked studies. Study 1 assessed reliability using Cronbach’s Alpha, Ordinal Alpha, Cohen’s Kappa, and ICC to evaluate internal consistency and rater agreement. Study 2 used differential item functioning (DIF) analyses with logistic and ordinal regression to examine scoring biases across race, gender identity, veteran status, age, and disability. Study 3 used Kaplan-Meier estimators and Cox proportional hazards models to evaluate whether SPDAT scores aligned with actual housing outcomes and shelter returns, both with and without housing assistance (Rapid Rehousing [RRH], Permanent Supportive Housing [PSH]).

Results:
The SPDAT and F-SPDAT demonstrated higher internal consistency and rater reliability than the VI-SPDAT tools. Reassessments by the same assessor produced more consistent results than those by different assessors, though reliability declined notably after six months—reflecting the evolving nature of homeless vulnerability over time. DIF analyses revealed consistent scoring disparities across the SPDATs: BIPOC individuals, men, older adults, and Veterans received lower vulnerability scores than their White, female, younger, and non-Veteran counterparts, despite often facing elevated risk. These scoring disparities became even more consequential when paired with survival analysis findings. PEH who did not receive a housing voucher returned to shelter within a median of 28 days, while those receiving RRH and PSH remained out of shelter for 841 and 1,522 days, respectively. These interventions reduced shelter return risk by 65% and 72%. Notably, groups who scored lower in the DIF analysis—particularly BIPOC individuals—also had one of the highest shelter return rates when not housed, revealing a disconnect between assessed scores and actual need.

Conclusions and Implications:
This study provides strong evidence that current SPDAT-based tools may systematically under-prioritize key demographic groups, leading to missed housing opportunities and increased shelter recidivism for underprioritized populations. Results underscore the importance of frequent reassessments, continuity in assessor-client relationships, and the development of homeless vulnerability tools that are culturally responsive and psychometrically sound. Methodologies such as DIF and survival analysis offer valuable frameworks for evaluating tool fairness and predictive utility. Future assessments should integrate administrative data and adaptive technologies such as Computerized Adaptive Assessments (CAA) to improve precision, equity, and impact in homelessness service systems.