Assessing the Reliability of the Women's Risk Needs Assessment within an Alabama State Women's Prison

Background & Purpose

Women and men entering the criminal justice system bring different histories and experiences. To provide female inmates with equitable care, scholars and policymakers have increasingly examined gender-specific risks and needs. These efforts have prompted implementation of gender-responsive classification tools within US prison systems, including the Alabama Department of Corrections (ADOC). In 2018, the ADOC implemented a gender-responsive assessment tool designed to guide classification and programing within the prisons – the Women’s Risk Needs Assessment (WRNA). Through a four-year, multi-site study, researchers from three institutions, alongside ADOC staff, are validating the WRNA. This interdisciplinary effort is the first in the nation to examine validity of the WRNA within a statewide prison system.

Before researchers began validation of the WRNA, a study of reliability was conducted. Reliability is an essential component of the risk assessment process, as inconsistent administration jeopardizes the predictive value of assessments. This study tests the reliability of the Women’s Risk Needs Assessment within an Alabama state women’s prison.

Methods

Two studies of reliability were conducted within the intake sample (n=30) of women entering an Alabama state prison. Women were eligible for inclusion if they were expected to serve a sentence of six months or longer. Study 1 assessed inter-rater reliability though an initial video-recorded administration of the WRNA (Time-1; n=27) by classification specialists (n=6). Video-recordings captured classification specialists in frame, with inmates’ responses to the WRNA documented through audio. Time-1 video-recordings were reviewed by a separate classification specialist one to three weeks after the Time-1 administration (n=26). Study 2 assessed test-retest reliability through a second video-recorded administration of the WRNA (Time-2; n=27) by a different case specialist approximately three weeks after the Time-1 administration (n=27).

Within both reliability studies, video-recordings were audited to assess systematic challenges to WRNA administration. Cohen’s Kappa indicated percentage of rater agreement, where Kappa > 0.70 indicated suitable inter-rater and test-retest reliability across items and aggregate scores. Data were analyzed separately by two researchers within separate software packages – SPSS (v25) and Stata (v16) – to ensure accuracy of findings.

Results

The inter-rater and test-retest reliability analyses show agreement higher than 70% (Kappa > 0.70), which indicates reliable administration. The inter-rater reliability study yielded a Kappa score of 0.87 (87% agreement). The test-retest reliability study yielded a Kappa score of 0.89 (89% agreement). The results indicated agreement among different WRNA raters and consistency in the inmates’ responses from Time-1 and Time-2.

While inter-rater and test-retest reliability results were strong, video-recordings revealed inconsistencies that could challenge reliable WRNA administration, including completing assessment-related case file reviews post-hoc (WRNA recommends an a-priori approach) and posing WRNA questions out-of-sequence.

Conclusions and Implications

Findings suggest that, despite minor inconsistencies noted within video-recordings, the WRNA is administered consistently across raters at this Alabama state women’s prison. Additionally, classification scores remain stable over the first several weeks following intake. These processes and findings provide practical guidance and may support implementation of gender-responsive tools within other statewide prison systems across the country.