What to do When the “Gold Standard” Can't be Realized: The Use of Propensity Scores as a Way to Address Attrition in Random Experiments

When it comes to evaluating the effectiveness of interventions, the random experiment is considered the “gold standard.” Randomization is considered the gold standard because it provides a way of decreasing the chance that systematic differences, other than type of intervention, will obtain between the treatment and comparison group (s). The problem with random assignment, however, from the perspective of social work researchers is that, due to ethical and other reasons, it is often not feasible. Statisticians (Rubin, 1997) have developed a method to address this problem, called propensity score matching, and a few social workers have used this approach in their work (Koh and Testa, 2008 and Barth, Guo, and McCrae, 2008). According to statisticians, if a researcher, faced with observational data, matches study participants on propensity scores before evaluating the effectiveness of an intervention, this can go a long way toward addressing the problems posed by lack of random assignment. But what has received little attention in the literature is the use of propensity scores along with random experiments. Even with randomization, researchers may end up facing some of the same problems faced when dealing with observational data because of attrition. That is, those who drop out of the study may do so in such a way that the intervention and comparison groups are no longer “balanced” in terms of values on confounding variables. This is essentially a missing data problem. Two of the more popular techniques to address such problems are Maximum Likelihood and Multiple Imputation. But it's impossible to determine if the assumptions on which these techniques are based hold, even approximately so (Allison, 2002). Thus, it's questionable whether social work researchers should rely on such methods to handle attrition. After discussing the shortcomings of Maximum Likelihood and Multiple Imputation, my paper will discuss how propensity scores can be used in conjunction with multiple regression analysis to handle attrition in random experiments. This will be done by delving a bit into the mathematics of multiple regression analysis in an effort to indicate just how it “controls for” some variables, while isolating the effects of others.

References

Allsion, Paul (2002). Missing Data. California: Sage Publications.

Barth, Richard P.; Guo, Shenyang; and McCrae, Julie S. (2008). “Propensity Score Matching Strategies for Evaluating the Success of Child and Family Service Programs.” Research on Social Work Practice, vol. 18, no. 3, p. 212-222.

Koh, Eun and Testa, Mark F. (2008). “Propensity Score Matching of Children in Kinship and Nonkinship Foster Care: Do Permanency Outcomes Still Differ?” Social Work Research, vol. 32, no. 2, p. 105-116.

Rubin, Donald B. (1997). “Estimating Causal Effects from Large Data Sets Using Propensity Scores.” Annals of Internal Medicine, vol. 127, issue 5, p. 757-763.