Hidden Selection Bias and Rosenbaum’s Sensitivity Analysis

Purpose: Selection bias is a central challenge to causal inference and program evaluation. Over the past two decades, advanced statistical models (e.g., propensity score matching, optimal matching, etc.) have been developed to correct for overt selection bias due to researchers' failure of using available variables. However, none of these new models can handle hidden selection bias due to uncollected or unmeasured heterogeneity. Using the newly developed approach by Rosenbaum (2002, 2005), this workshop aims to address two questions: What are the challenges that researchers face when doing observational studies in which hidden selection bias is present? How might a researcher assess the sensitivity of findings to hidden selection bias?

Contents: The workshop begins with a discussion of the sources, types, and consequences of selection effects on causal inference. Hidden selection is virtually not a problem in classical randomized clinical trials, because the mechanism of randomization balances data for both measured and unmeasured variables to make the average of omitted variables equal to zero. Researchers using nonexperimental data should make extra effort to deal with hidden selection bias.

It then describes the Rosenbaum's approach of sensitivity analysis. The core idea of such approach is to use existing data to estimate the level of bias. Although selections are hidden and in a black box, researchers may estimate the sensitivity of study findings to hidden selections, and gain sufficient information for improving the study in future. Using a well-designed spreadsheet, the workshop illustrates the basic idea for one of the Rosenbaum's models (i.e., the Wilcoxon's signed-rank test for sensitivity analysis of a matched pair study). It also demonstrates the implementation of such analysis using Stata rbounds program.

It then illustrates the application of the sensitivity analysis to a social work study evaluating the impact of poverty on children's academic achievement. Using the 1997 national survey data of Child Development Supplement, the study conducted an optimal matching using six covariates (i.e., caregiver's education, caregiver's use of AFDC in childhood, and demographic variables), and found that in 1997, children who used AFDC had an average letter-word identification score that was 3.17 points lower than that of children who never used AFDC (p<.05). Using Rosenbaum's approach, the sensitivity analysis shows that the study is sensitive to hidden bias at a sensitivity level of 1.43. Because 1.43 is a small value, the analysis reveals that the study finding is very sensitive to hidden bias, and therefore, further analysis that controls for additional biases is warranted.

The workshop concludes by summarizing potential areas in social work research that should routinely consider using the sensitivity analysis to discern the level of hidden selection bias.

Pedagogical Techniques: Teaching methods include lecture, PowerPoint presentation, and Excel/Stata demonstration.