Bridging Disciplinary Boundaries (January 11 - 14, 2007)

Purpose: This study sought to assess the extent to which the reportage of published outcome studies risks misleading practitioners engaged in the evidence-based practice (EBP) search process as to the effectiveness of evaluated interventions, programs, or policies. Methods: All articles (N=136) reporting outcome studies from 2000-2005 found in two social work research journals and two PsycINFO database searches (regarding domestic violence and child maltreatment interventions) were assessed with regard to research design, findings and appropriateness of causal inferences. To assess inter-rater reliability, the authors of this paper independently (blind to each other's data) recorded the desired information for 26 randomly selected articles. Inter-rater reliability was good at 96.2% for design, 100% for finding, and 77% for appropriateness of causal inferences. Results: The majority of articles (69%) used designs that do not warrant making conclusive causal inferences, and the authors of most of these articles (62%) made problematic causal inferences that could mislead practitioners who are rusty on research methods or who do not scrutinize the articles carefully. Of particular concern was the finding that 80% of the authors of articles reporting pre-experimental designs made problematic causal inferences. There were not statistically significant differences between the social work research journal and PsycINFO sub-samples in terms of the proportions of studies making appropriate and inappropriate causal inferences (X²=3.405, df=2, p=.182). Implications for Research and Practice: Implications are drawn regarding the following: 1) How these findings might affect practitioners that engage in evidence-based practice and the evidence-based practice movement in general; and 2) Recommendations for changes in journal review processes, research dissemination to the practice community, and researchers who seek to publish in the future.