Research That Matters (January 17 - 20, 2008)

Purpose: Surveys are integral to social work research, as often the foci of study are perceptions, attitudes, and private behaviors. Yet, as evidenced in the social work journals, survey researchers often settle for poor response rates, justifying their methods by arguing that their samples are representative of their population. The purpose of this analysis was to explore whether conducting multiple steps of data collection (e.g., Dillman's Tailored Design Method) that lead to higher response rates result in differing explanatory models when analyzing data intended to inform social work interventions.

Methods: We utilized a cross-sectional anonymous mailed survey design to collect data for the original survey. Using a systematic random sampling strategy, we chose a sample of 1000 actively practicing NASW members from North Carolina and surveyed them about health and occupational issues. Next, we identified four timeframes that corresponded to our structured mailing schedule and distribution of returned questionnaires, and we calculated response rates for each (37.7%, 57.7%, 68.7%, and 75.1%). We filtered the data by these response rates, and then conducted a purely empirical regression analysis on each of the four groups, using depressive symptoms as the outcome variable and a pool of personal and occupational independent variables that were culled from the literature and significant in a bivariate analysis.

Results: We found no differences among the four sample groups on demographic variables or in prevalence of depressive symptoms, and the samples were demographically representative of the population of NASW members. Despite this, the statistical models were quite different. Altogether, six independent variables were significant in all four analyses, but five additional independent variables were significant in some, but not all the analyses. The 37.7% response rate model included six IVs, the 57.7% response rate had eight IVs, and the higher response rates included ten significant IVs. The most influential variable in first analysis was fifth most influential by the final analysis, and the least influential in the first analysis was most important in the final analysis. Beta weight magnitudes varied by as much as 65% across the four analyses.

Implications for practice: The results of this analysis clearly illustrate the importance of achieving high response rates for survey studies. Even though the demographics and the outcome variables showed no differences among larger and smaller samples, the findings from each response rate group differed in the variables that were significant in the analysis as well as the magnitude and rank order of influence of the beta weights. The typical method of examining whether the demographics of the sample are representative of the population is not, by itself, a useful strategy – especially if the demographics are not significantly related to the independent variable. This is of critical importance for studies that are intended to inform interventions. If the data collection in this study had ceased after the first or second mailing, the variables targeted for intervention would be very different from those that would be targeted after multiple data collection waves.