Investigating Patterns in Practice Data to Evaluate What Interventions Work and for Whom: Binary Logistic, Multinomial Logistic Regression Models, and Receiver Operating Curves

This workshop will present an introduction to applied categorical data analyses (binary and multinomial logistic regression), when the outcome has three or more ordered categories (ordinal multinomial logistic regression), and where the outcome is a continuous scale (receiver operating curves) (Hosmer & Lemeshow, 2000; Kazi, 2003; Jaccard & Dodge, 2004). The presenters will use datasets from their completed evaluations from New York and United Kingdom, and discuss real-world applications of the analyses. The didactic approach will be interactive, guiding the workshop participants through the requirements and limitations of each method. Appropriate time for a question-and-answer period will also be included.

With regard to dichotomous outcome variables (e.g. those that improved and those that did not), binary logistic regression will be used to investigate what interventions work and in what circumstances. In each example, the variables that may be influencing the outcome will be identified through bivariate analysis and then entered in a forward-conditional model. The variables that are actually influencing the outcome are retained in the equation, and those that are significant provide an exponential beta indicating the odds of the intervention achieving the outcome where the significant factor(s) may be present. Nominal or unordered multinomial logistic regression will be explained using datasets that explored the outcome variable as clients who improved, stayed the same, or got worse from pre- to post-test. For example, with regard to driving while intoxicated datasets, offender status in terms of 1, 2, 3 or more arrests, and prior treatment for alcohol or other drug problems, of none, one, two, three or more will be explored. Variable selection, model fit strategies, and interpretation of the coefficients and odds ratios will be illustrated.

With regard to outcome scales with continuous measures, receiver operating curves (ROC Analyses) can be used to identify possible cutpoints where the scale of interest shows the beginnings of a differential relationship with the outcome of interest. The workshop will also consider how the identification of these points may be important for subsequent intervention planning and development. It is always difficult to show where a scale begins to have a more critical relationship with the outcomes. Regression discontinuity is one approach that is used with this in mind but where the cutpoint comes from is the question that will be addressed in the workshop. We also know from personality research that it is in the extremes where differences are generally found – it really comes down to what do we do with the group in the middle and how do we identify who in that middle group may be moving to the extreme score for negative consequences and who may be shifting downwards towards less negative impact –

Regression and ROC analyses can help researchers and practitioners understand outcomes, what predicts those outcomes, and characteristics of clients who are achieving those outcomes verses clients who are not; but technical assistance is needed to provide social work researchers with the requisite skills and knowledge to carry out these often useful but underutilized analyses.