Improving Propensity Score Analysis Using Machine Learning: A Friendly Introduction with R Demonstration

Propensity score (PS) methods (Guo, & Fraser, 2015) have experienced a tremendous increase of interest in many social science areas including social work. Propensity score methods estimate a conditional probability that expresses how likely a participant is to be assigned to or to select the treatment condition given certain baseline characteristics. By conditioning on the propensity score, the goal is to achieve balance on the observed covariates and recreate a situation that would have been expected in a randomized experiment (Thoemmes, & Kim, 2011). Holding certain assumptions, a PS analysis can yield unbiased causal effect estimates in observational studies.

While PS methods have advanced significantly over the past decades, there remains heated debates on PS methods' dis/advantages in reducing selection bias and effective estimation of treatment effects, among others (Pearl, Glymour, & Jewell, 2016; Ali et al., 2015). One major critique is that parametric models (e.g., logistic regression) that are commonly used to estimate PSs require assumptions of variable selection (strong ignorability), the functional form and distributions of these variables, and specifications of interactions (Shadish, 2013). If any of these assumptions, that often cannot be empirically examined, are not met, covariate balance may not be achieved by conditioning on the propensity score, and will result in biased treatment effect estimate.

Advancements in statistical machine learning (James, 2013) over the past decades have offered alternative PS estimation methods that are arguably superior to existing parametric models (Linden, & Yarnold, 2016). One such method is the Classification and Regression Tree (CART) methods (Loh, 2008) which refers to a diverse number of classification and prediction algorithms. Contrary to statistical approaches to modeling that assumes a data model with parameters estimated from the data, CART tries to extract the relationship between an outcome and predictor(s) through a learning algorithm without a priori data model (assumptions) (Shadish, & Steiner, 2010). CART methods well complement challenges facing existing PS estimations that require multiple assumptions (Diamond, & Sekhon, 2013). More importantly, because PS methods aim to achieve covariate balance rather than estimating a true relationship, empirical studies have supported CART methods being more effective in achieving this goal, especially when the true relationship (between predictor(s) and propensity score) is complicated (Lee et al., 2010). Additional advantages of CART in estimating PS include: well handles categorical, ordinal, continuous, and missing data; insensitive to outliers and monotonic transformations of variables, and naturally models interactions and nonlinearities (Lee et al., 2010).

Workshop presenters will start with an overview of existing PS estimation methods, their challenges, and then explain the need for and demonstrate the use of the CART methods in estimating PSs. Material will be provided that will include (1)a conceptual overview of CART methods in the context of PS analysis; (2)examples of implementing basic and advanced/ensemble-based CART methods in PS analysis. This 90-minute workshop is designed to help participants: (1)understand the critiques of existing PS methods; (2)understand the concepts and advantages of CART methods; (3)implement CART-based PS estimation; (4)evaluate improvements using CART-based PS estimation versus traditional approaches.