Developing a Reliable Assessment Tool for Refugee Survivors of Torture

Background:  Research estimates that around 44% of refugees in the U.S. have experienced torture (Center for Victims of Torture, 2015). Applying this percentage to the more than three million refugees who have resettled since 1975, the number of refugee survivors of torture (SOT) could be as high as 1.3 million, or even higher, since this estimate does not account for SOT granted a legal asylum protection.  As refugees, SOT have to overcome adversities posed by physical and psychological sequelae of trauma compounded by environmental difficulties associated with resettlement, acculturation, and striving for self-sufficiency (Kira, 2002). Rehabilitation of SOT requires a coordinated system of care that integrates multi-disciplinary knowledge of human behavior with compassionate service delivery (Kira, 2004). However, assessment tools that address the multisystemic needs of SOT are few, and, if they exist (Barneche & Matos, 2014), tend to focus primarily on mental health (Montgomery & Patel, 2011), do not assess environmental factors (Kinzie & Jaranson, 2001), and lack holistic integration (Jaranson & Quiroga, 2011). To mitigate this gap in research, practice, and policy, the Survivors of Torture-Psychosocial Well-being Index (SOT-PWI) was developed to serve as a new tool for assessment and successfully piloted (Hodges-Wu & Zajicek-Farber, 2017). The current study sought to further investigate the inter-observer reliability of torture-treatment providers using the SOT-PWI, and was guided by three research questions: (1) Were providers able to consistently assess different SOT-clients across multiple areas of well-being? (2) Did providers’ education levels influence the reliability assessments? (3) Did providers’ work-experience levels also influence the reliability assessments? This study was approved by authors’ university institutional review board.

Methodology: The study implemented a cross-sectional survey design by recruiting professional torture- treatment providers through the National Consortium of Torture Treatment Programs, with 71% response-rate. Procedures required providers to read a training guide about the SOT-PWI, practice their assessments before applying the tool following a systematic protocol, and anonymously rate-assess four case-vignettes. The vignettes described various SOT-client characteristics and severity of needs.  Analyses employed intra-class correlation coefficient (ICC) (Landers, 2015) with a two-way fixed model with average absolute agreement for ratings (Richmond et al., 2015).

Results: Participants included 47 service providers, of whom 45% completed college degrees and 55% graduate education.  Forty percent had low-level, 22% medium-level, and 38% high-level work-experience. The inter-observer reliability findings revealed robust strength ICC of 0.98 (CI 0.97-0.99) for all participants across all cases and domains. For undergraduate and graduate education, ICC findings were equally robust. Strong ICC findings were also seen on the three levels of work experience. However, when examining ICC results by increased complexity of cases, findings revealed that lower levels of education and work-experience compromised providers’ consistency. Further ICC results address short version of SOT-PWI, study limitations, and future research.  

Implications: The SOT-PWI offers providers a reliable assessment instrument for documenting, evaluating, and addressing the holistic needs of SOT populations. Experiences derived from education and direct practice stand to influence provider consistency in tool application and need to be supported by professional practice training and development.