Using a Directed Acyclic Graph to Inform Research with Propensity Score Analysis

Background and purpose: Social work researchers seeking to estimate the causal effect of non-randomly assigned treatment frequently utilize propensity score analysis (PSA) to balance the distributions of confounding variables among treated and untreated participants. Given that a key assumption in PSA is that measured variables account for the influence of unmeasured variables, researchers may naively include all possible confounders into the adjustment stage with little consideration of the associations between confounders, the treatment, and the outcome. This may induce bias, particularly if variables only associated with the treatment and not the outcome are used. Directed Acyclic Graphs (DAGs) provide a visual representation of the causal relationships between variables as informed by theory and existing literature, can be used to ensure PSA is the best method in a given context, and can guide selection of predictors in both stages of PSA. In this presentation, we present an example of a PSA study in which a DAG was essential to making these decisions. This is an application of the third step of using a DAG to inform social work research design and analysis.

Methods: This exemplar study used a quasi-experimental study design to estimate the causal effect of student gatekeeper training (i.e., treatment) on their use of counseling services for mental health problems (i.e., outcome). A DAG was built to visualize the association between gatekeeper training, counseling utilization and a set of confounders according to theory and empirical studies. This visualized tool informed PSA in three ways: (1) no other method (e.g., instrumental variables) seemed adequate for estimating an unbiased effect; (2) the selection of predictors of the propensity score model was informed by a DAG which shows an extensive list of correlates of counseling utilization (e.g., gender or mental illness diagnosis); (3) the DAG was used to identify the minimal sets of covariates that needed to be included in the outcome model. Secondary data was retrieved from the Healthy Mind study 2020-2021, a cross-sectional study investigating mental health status, service utilization, and related factors among university students.

Results: We used the propensity score model, guided by the DAG, to predict the receipt of gatekeeper training, which balanced the treatment group and comparison group across a set of confounders. Finally, to answer our research question, a logistic regression model adjusting for covariates was conducted by using propensity score weighting. By using PSA, we found that receiving student gatekeeper training was associated with 19% higher odds of counseling utilization compared to students who did not receive gatekeeper training.

Conclusion and implications: PSA is widely used throughout social work research to estimate unbiased causal estimates. The risk of inducing bias is ever present, however. The DAG framework can be used to rule out other approaches for analysis, and then to subsequently improve PSA studies by emphasizing good predictor selection for both the propensity score and outcome stages of analysis. By using this approach, researchers may be able to reduce the risk of inducing bias, improving the credibility of causal estimates obtained using PSA.