Examining the Sensitivity and Specificity of DSM-5 Categories

Background: A psychosocial interview utilizing DSM-5 is the standard method for determining whether a client has a mental disorder or not. Thus, one of the most important issues regarding the DSM-5 is its diagnostic accuracy. 

Purpose: To examine the accuracy of DSM-5 diagnoses through re-analyses of published field trial data which focus on sensitivity and specificity. 

Method: The Kappa statistic is commonly used to measure inter-rater or test-retest agreement in field trials of psychiatric diagnoses. The published data on the reliability of DSM-5 primarily reports kappa statistics. Kappa may be difficult to interpret for clinicians, and sensitivity and specificity statistics may be preferable. Feurman and Miller (2005, 2008) have proven the mathematical relationship between kappa and sensitivity/specificity. We used their formula to derive sensitivity x specificity curves from the kappa values found in the DSM-5 field trials, as well as Negative and Positive Predictive Values (NPV/PPV). 

Results: Graphical analysis will be presented for all DSM-5 categories tested in the field trials. The modest kappa values from the DSM-5 field trials result in undesireable sensitivity x specificity curves for some categories. This is particularly true if sensitivity and specificity are both treated as equally important. For example, in Generalized Anxiety Disorder (GAD), the maxima for sensitivity/specificity is .60, which results in a PPV of 0.27. At the clinic prevalence rate of GAD, this means that 32% of identified cases would be false-positives. Under the same assumptions, other disorders such as Post-Traumatic Stress Disorder (PTSD) and Attention-Deficit Hyperactivity Disorder (ADHD) had higher PPVs (e.g., for ADHD, 0.87). NPV values were relatively high when sensitivity=specificity, and acceptable in general. 

Implications: According to our analysis, the DSM-5 field trials found that well-trained clinicians can rule out mental disorder (high NPV) much more effectively than accurately diagnosing the presence of a mental disorder. The DSM-5 criteria for some of the most common mental disorders (e.g., GAD) did not perform well as a diagnostic test. This is important because the DSM-5 field trial clinicians were specially trained and used assessment instruments. Unfortunately, practice is less consistent in the community and therefore DSM-5 diagnoses are likely to be even less accurate than presented in the field trials. Thus, efforts to improve diagnosis and treatment might benefit from integrating these findings into both practice and research. That is, the lack of reliability of DSM-5 categories potentially opens up new lines of both practice and research. Research efforts might focus on improving the diagnostic accuracy of the DSM-5, while practice settings might investigate alternatives to conventional diagnosis. Finally, these findings raise questions about the adequacy of the present informed consent process when diagnosing clients with a DSM-5-defined mental disorder.   

Feuerman, M., & Miller, A. R. (2005). The kappa statistic as a function of sensitivity and specificity. International Journal of Mathematical Education in Science and Technology, 36(5), 517-527.

Feuerman, M., & Miller, A. R. (2008). Relationships between statistical measures of agreement: sensitivity, specificity and kappa. Journal of Evaluation in Clinical Practice, 14(5), 930-933.