Abstract: Choosing Covariates and Evaluating Balance in Propensity Score Methods: A Primer Based on Recent Recommendations from the Methodological Literature (Society for Social Work and Research 21st Annual Conference - Ensure Healthy Development for all Youth)

Schedule:

Friday, January 13, 2017: 8:00 AM

La Galeries 2 (New Orleans Marriott)

* noted as presenting author

Kirsten Kainz, PhD, Associate Professor, University of North Carolina at Chapel Hill, Chapel hill, NC

Noah Greifer, Student, University of North Carolina at Chapel Hill, Chapel hill, NC

Ashley Givens, MSW, Doctoral Student, University of North Carolina at Chapel Hill, Chapel Hill, NC

Susannah Zietz, Student, University of North Carolina at Chapel Hill, Chapel Hill, NC

Karen Swietek, Student, University of North Carolina at Chapel Hill, Chapel Hill, NC

Brianna M. Lombardi, MSW, Doctoral Student, University of North Carolina at Chapel Hill, Chapel Hill, NC

Jamie Kohn, Student, University of North Carolina at Chapel Hill, Chapel Hill, NC

Background/Purpose

Propensity score methods are frequently used quasi-experimental tools that reduce the bias in estimated statistical relations due to confounding when exposure is non-randomly assigned. For propensity score methods to reduce bias due to confounding they must approximate experimental designs where all pre-exposure characteristics of the study participants, including their potential responses to exposure, are unrelated to exposure assignment and therefore ignorable when estimating effects. This is known as the strong ignorability (SI) assumption. To meet the SI assumption, the full set of confounders - those that predict selection into exposure and those associated with variation in the outcome - must be included in the propensity score model. Further, the propensity score model must yield exchangeable groups via weighting, stratification, or matching on the propensity score, where the exposed and non-exposed groups are balanced on all measured confounders.

It is not possible to prove that the full set of confounders has been included in the propensity score model. Rather, researchers must defend their propensity score models with two arguments: 1) a review of theory and evidence justifies that the selection of covariates for the estimation of propensity scores approaches the full set of confounders; and 2) statistical evaluation of covariate balance across the exposed and non-exposed groups following matching, stratification, or weighting justifies the propensity score model. Recent systematic reviews of propensity score analyses published in social work and public health journals indicate that in many cases researchers fail to provide these justifications, despite increasing methodological recommendations to guide practice. Because the justifications are essential for evaluating the merit of propensity score methods to reduce bias due to confounding we offer this paper as a guide for applied researchers who wish to incorporate currently recommended practice for selecting covariates and evaluating balance in their research using propensity score methods.

Methods

In this paper we move from theory to applied examples using R, citing published recommendations for good practice in the selection of covariates and evaluation of covariate balance. We analyze data from the Early Childhood Longitudinal Study Kindergarten Cohort of 2011 to provide concrete examples of analytic techniques and interpretations.

Results

We demonstrate that different propensity score models yield different covariate balance results, and model specification and matching techniques can be adjusted to improve balance across models. The final propensity score model should be chosen based on evidence of superior balance.

Conclusions

Researchers using propensity score methods will improve their analytic results and inferences by closely following published recommendations for covariate selection and balance implied by the SI assumption. Recent systematic reviews and simulation studies provide updated guidelines for covariate selection and balance evaluation. Recommendations and examples provided in this paper are intended to motivate defensible analytic practice, encourage thoughtful reviews of research, and ultimately produce a strong research base from which to design social work policy and practice.