What Works and for Whom: Combining Realist Evaluation, Effectiveness Research and Epidemiology Traditions in Social Work and Other Human Services Utilizing Data from Management Information Systems: Demonstration with Real Whole School Data

This is a workshop demonstration of award-winning realist evaluation with live data analysis of real big data from the largest school district in Chautauqua County (NY). Research methods drawn from both epidemiology and effectiveness research traditions are demonstrated in a realist evaluation in partnership with human service agencies and the school district to investigate what programs of intervention work and for whom. Real live data from management information systems (schools, social services, mental health, youth justice) is used to investigate the effectiveness of the human service interventions. As the emphasis is on data naturally drawn from practice, quasi-experimental designs will be demonstrated using demographic variables to match intervention and non-intervention groups. Binary logistic and linear regression will be demonstrated as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence. Evaluators and agencies can make the best use of the available data to inform practice. Realist evaluation essentially involves the systematic analysis of data on 1) the service users’ circumstances; 2) the dosage, duration and frequency of each intervention in relation to each user; and 3) the repeated use of reliable outcome measures with each service user. The workshop will show how evaluators work in partnership with these agencies, to clean the data, undertake data analysis with them at regular intervals and not just at the end of the year. In this way, the evaluators and the human service agencies can work together to evaluate the impact of interventions on the desired outcomes utilizing innovative methods and addressing issues relevant for practice including diversity, investigating where and with whom the interventions are more or less effective in real time. Establishing cause and effect is a particular theme of this demonstration. As the data mining includes all service users (e.g. all school children in school districts), it is possible to investigate the differences in outcomes between intervention and nonintervention groups, and these groups can be matched using the demographic and contextual data. The innovative methods demonstrated using the same data would include those that are part of the family of methods used to determine epidemiologic evidence based on association, environmental equivalence, and population equivalence. For example, the presenters will use datasets from their completed evaluations from New York State, and discuss real-world applications of the analyses. The didactic approach will be interactive, guiding the workshop participants through the requirements and limitations of each method. Binary logistic regression will be used to investigate what interventions work and in what circumstances. In each example, the variables that may be influencing the outcome will be identified through bivariate analysis and then entered in a forward-conditional model. The variables that are actually influencing the outcome are retained in the equation, and those that are significant provide an exponential beta indicating the odds of the intervention achieving the outcome where the significant factor(s) may be present. The interactive live demonstration will investigate where an intervention is more or less likely to be effective, and how to utilize findings and inform practice on demand.