Assessing the Reliability of Self-Reports of Criminal and Illicit Activity

Background: Self-reports are frequently used in studies of criminal and illicit behavior. Any study that is reliant on self-reported data must be critically concerned with the accuracy of those data, as reporting biases or errors in self-reported data can threaten study validity. It is possible, for example, that when asked about sensitive topics, such as criminal history, participants may be reluctant to divulge information to a researcher and thus severely underreport on this measure. Within the context of a randomized trial, participants in more intensive treatment arms may feel greater rapport with research and intervention staff, and may thus be more willing to provide candid responses. Participants may also provide inaccurate answers for other reasons, including constraints on their memory or due to lack of interest in the study. The current study is aimed at examining the multifaceted relationship between trust in interviewers and the accuracy of self-reported data, as well as the accuracy of reporting crimes and illicit behaviors using official/administrative records.

ROMI Project: Reducing Opioid Mortality in Illinois (ROMI) is a NIDA-funded randomized intervention trial that seeks to evaluate the effect of linkage services designed to help recently incarcerated persons living with substance use disorders to connect with addiction treatment and harm reduction services. Key data collected in this study are self-reported and are provided during interviews with researchers.

Methods: We compare self-reported data from a sample of 144 ROMI participants to official records from the Illinois Criminal Justice Information Authority (ICJIA) to assess the reliability of our participants’ self-reports. We evaluate the concordance between these datasets in terms of key justice outcomes, such as arrests and illicit activities. After demonstrating the general reliability of our data, we also provide specific insights into the interview and data collection process that was used in gathering these self-reported data, providing an ethnographic analysis of our interviewers’ practices. We qualitatively assess which practices are the most effective in building rapport and generating accurate self-reports.

Results: Participants slightly over-report past criminal histories and criminal activities relative to data available in administrative records. This pattern reflects the reality that official records can only provide information on activities for which an arrest was made, and this is sure to exclude a portion of the criminal activity most participants were likely involved in. In this way, the use of self-reported data may provide additional benefits beyond what official records can provide.

Conclusions and Implications: Given the high degree of concordance between self-reporting and official data, these patterns increased our confidence in the reliability of our self-reported data. This is likely thanks in large part to the good interview practices of our research assistants, who gained the trust of and built consistent rapport with our study participants. Taken together, our study findings further stress the need to have trained field workers who have expertise in engaging study subjects in order to obtain reliable self-reported data.