In Pursuit of Quality: A Review of Quantitative Studies Published in Social Work Journals

Background and Purpose

The social work profession has grappled with the nature, extent and quality of its research enterprise for over a hundred years. Today, the importance of rigorous research to the profession is widely-accepted. Some contemporary studies have sought to gauge the rigor of social work research and a few have done so by examining published articles in social work journals.  This work was conducted to describe the methods employed in quantitative social work articles and to assess the extent to which these methods conform to widely-accepted standards of quality.

Methods

We drew a proportional stratified random sample of 120 articles describing quantitative research published between 2010 and 2015 in social work journals. Then two senior researchers independently reviewed the articles using a 16-page template with dichotomous measures to gauge the presence of quality indicators. Inter-rater reliability was assessed and the resulting Kappa (.70) indicated “substantial” agreement between reviewers on categorical measures. After collecting the template data we used Google Scholar to check the number of citations for each article.

A “quality index” measure was constructed from eight dichotomous questions and a “reviewer evaluation,” consisted of one question, “How would you evaluate this article?” with five response options ranging from “poor” to “excellent.”

This presentation reports on descriptive and bivariate statistical analyses related to the quality indicators.

Results

The use of experimental and quasi-experimental designs was rare (6%), while a majority (63%) of the studies reported cross-sectional and descriptive designs. Non-probability sampling was the norm (71%), and the dominant sources of data were interviews (36%) and written surveys (34%).

Only one of the dichotomous quality indicators was used in a majority of articles: 77% addressed the limitations of the study. Just 15% described how missing data were handled and less than half of articles reported affirmatively on any of the quality measures employed here. Over half (60%) of the articles contained no reference to theory.

Authors who had PhDs scored significantly higher on the quality index than those without (p<.05) and were more likely to specify hypotheses. Likewise, PhD authors were significantly more likely to address the validity of their measures (p<.05).

We found no significant association between citations and the quality index; however, there was a weak correlation (R=.254; p<.01) between reviewer evaluation and citations.

Conclusions and Implications

The widespread failure to address theory, power, statistical assumptions, missing data and other quality indicators in these articles might reflect the indifference of social work editors and reviewers; or, it may indicate that social work researchers use sloppy methods. A more rigorous review process in social work journals could promote greater methodological rigor. Our findings suggest that doctoral programs may indeed contribute to the quality of the profession's knowledge base. They also call into question the use of citations in the promotion and tenure process as indicators of the quality of scholarly contributions.