Abstract: New Methods in Randomized Controlled Trials: Using SMS and Mixed-Methods to Evaluate a Guaranteed Income Experiment (Society for Social Work and Research 24th Annual Conference - Reducing Racial and Economic Inequality)

New Methods in Randomized Controlled Trials: Using SMS and Mixed-Methods to Evaluate a Guaranteed Income Experiment

Schedule:
Friday, January 17, 2020
Independence BR H, ML 4 (Marriott Marquis Washington DC)
* noted as presenting author
Stacia Martin-West, PhD, Assistant Professor, University of Tennessee, Knoxville, Nashville, TN
Amy Baker, PhD, Assistant Professor, University of Pennsylvania, Philadelphia, PA
Mina Addo, MS, Doctoral Student, University of Pennsylvania
Daniel Horn, MSW, Graduate Research Assistant/Ph.D. Student, University of Tennessee, Knoxville, TN
Stacy Elliott, MSW, Graduate Research Assistant, University of Tennessee, Knoxville, Knoxville, TN
Background

The Stockton Economic Empowerment Demonstration (SEED) is the country’s first city-led guaranteed income (GI) pilot. In 2019, SEED began providing 125 Stocktonians with a GI of $500 per month for 18 months. The income is distributed monthly through prepaid debit cards. Since the income is “guaranteed,” there are no work requirements or restrictions on how the money can be spent.

Context

SEED has been rooted in a replicable, evidence-based public policy model that answers Durr’s (1993) question, “What moves policy sentiment?” To that end, the intervention, research, and dissemination strategy were conceptually designed to detect policy-relevant effect sizes while privileging community voices in evidence-based policymaking (Urban Institute, 2018). This approach recognizes that public impact rests on translating empirical evidence into narrative and storytelling formats capable of shifting moods across varying stakeholders (Kingdon, 2011; Jenkins-Smith et. al., 2014).

Challenges and Opportunities

To answer the research questions, (1) How does GI impact monthly income volatility? (2) To what degree do changes in income volatility alter inequity drivers and social determinants of health? (3) How does GI generate agency over one’s future?, novel design and data collection methods have been implemented. We implemented traditional RCT methods, using an addressed based random sample to select 478 households. Random assignment was used to designate 125 participants to treatment, 117 to control, and 153 to an administrative control. However, the ability to answer these broad research questions rested upon overcoming two common obstacles to data collection and robustness in RCTs: participant response burden and lack of context to translate findings.

Innovations

With the exception of one study (Morduch & Siwicki, 2017), income volatility has been captured at six or twelve month intervals. This results in data that fails to adequately capture the frequency of income fluctuation. To overcome this obstacle, we built upon emergent methods of SMS or text-based surveys to collect monthly income volatility data over a 24 month period. Beginning in March 2019, respondents were asked to report their monthly income alongside additional measures of anxiety and mood. Response rates were at 88% across treatment and control.

To address the second obstacle of lack of context to translate the findings of RCTs, the research team has actively worked alongside the implementation team to document and problem-solve how the introduction of guaranteed income interacts with the social safety net. Further, the study is designed in two parallel mixed methods strands, such that as early signals of quantitative data emerge from each wave of surveys, samples of participants who contributed to early quantitative trends can be pulled into an in-depth interview cohort.

Implications

RCTs are rightly designed with considerable rigidity to capture the treatment effect. However, there exist tremendous opportunities to integrate mixed methods and novel data collection techniques that provide important context for policy innovation.