Combining Efficacy Research and Epidemiology Traditions: Utilizing Data Analysis Methods In 100% Natural Agency Data In Evaluating What Works and In What Circumstances

Social work researchers often use the effectiveness tradition (e.g. randomized controlled trials) to investigate whether outcomes are caused by social work interventions. Another tradition, epidemiology, concentrates on examining patterns in naturally occurring data to also establish causal connections, e.g. proving that smoking can cause lung cancer. Realist evaluation combines both of these traditions, utilizing naturally occurring agency data that includes at least one reliable outcome measure, to examine patterns in three sets of data: 1) demographics, 2) interventions, and 3) outcomes, to investigate what interventions work, for whom and in what circumstances.

One of the methods for improving practice within agencies may lie in the data that agencies collect as a part of their normal operations. While social service agencies continuously collect information, very seldom is it analyzed with the intent of improving the overall effectiveness of the services provided. The proposed workshop will focus on how these agencies might use these data to improve the effectiveness of client services.

This demonstration will illustrate how realist evaluation strategies can be applied in the evaluation of 100% natural samples in schools, health, youth justice and other human service agencies for youth and families. This demonstration will include new data analysis tools drawn from both the efficacy and epidemiology traditions to investigate patterns in this data in relation to outcomes, interventions and the contexts of practice. For example, binary logistic regression can be used repeatedly with whole school databases at every marking period to investigate the effectiveness of school-based interventions and their impact on school outcomes. The demonstration will include practice examples drawn from the SAMHSA funded System of Care that has enabled a 100% evaluation of over 40 agencies in Chautauqua County, New York State; and education, social work and youth justice services in Moray Council, Scotland. The discussion will include the importance of how data are collected, the consistency with which data are collected, and how the data can be analyzed. The presenters will use real data sets to demonstrate how existing data can provide insights into how practice might be altered to have a more positive impact on the clients.

The workshop will explore methods of assessing underlying dimensions of existing measures (exploratory and confirmatory factor analyses), and how identification of these underlying dimensions can help target specific areas for intervention modification or development. Another method that will be explored is survival analyses. In many instances, clients of social service agencies either relapse or recidivate. Survival analyses can be used to identify when hazards are high for specific subgroups of individuals. This can then lead to development of aftercare services that target specific subgroups with specific content for that subgroup. These approaches to research can address the twin problems of application of evidence-based practice and the evaluation of practice to investigate what works and in what contexts, providing regular analyses to inform practice as it unfolds.