Propensity Score Matching: King & Nielsen’s Critique and Discussion

Background and Purpose

Propensity score matching (PSM), an enormously popular method of processing data for causal inference, has been increasingly applied to social work research to address challenging research issues. However, the debate about its usefulness has never ceased since its birth when Rosenbaum and Rubin published their seminal paper in 1983. Most recently, Gary King and Richard Nielsen (2016) published a paper on their website entitled “Why propensity score should not be used for matching” and showed that the method “often accomplishes the opposite of its intended goal – increasing imbalance, inefficiency, model dependence, and bias.” In this controversial study, the authors criticize that Rosenbaum and Robin's proof about the properties of propensity score, although mathematically correct, is either of little use or misleading when applied to real data due to the following two reasons: (1) the theorem encourages researchers to settle for the lower standards of approximating only complete randomization and only average levels of imbalance, rather than a fully blocked experiment and balance in their own samples guaranteed to reduce model dependence; and (2) balancing only on propensity score does not balance the entire vector of covariates – in sample, equality between any two estimated scalar propensity scores does not imply that the two corresponding k-dimensional covariate vectors are matched exactly.

Given the popularity of using PSM in social work research and the sharply contradictory opinions among the methodologists, it is important to examine the key critiques made by King and Nielsen. This workshop attempts to accomplish this goal. It focuses on three core issues about using propensity score to balance covariates. Results of a Monte Carlo study are presented to support key points made by this presentation. Implications about the debates and caveats of applying PSM are offered at the end of the workshop.

Contents

The workshop will focus on three issues appeared to be critical in the current debate about the pros and cons of PSM: (1) a review of Rosenbaum and Robin's (1983) proof and a response to the critique that a mathematically correct proof is either of little use or misleading when applied to real data; (2) the implications of King and Nielsen's critique to the applications of propensity score to other types of corrections such as the optimal propensity score matching, the inverse probability of treatment weighting, and the propensity score subcalssification; and (3) problems with the Mahalanobis distance matching, a method King and Nielsen proposed to replace the nearest-neighbor within caliper matching.

Results of a Monte Carlo Study

A Monte Carlo study provides evidence that: (1) when important observed covariates are controlled for, a nearest-neighbor within caliper matching using propensity score provides equally optimal estimates as the Mahalanobis distance matching; and (2) reduction in sample size after matching is a nontrivial problem of the Mahalanobis distance matching.

Implications

The workshop concludes that the efficiency and unbiasedness of PSM really depends on the level of controlling for hidden selections; when important covariates are omitted, all matching methods run into problems.