Dosage Analysis in a Controlled Trial of the Making Choices Program

Background and Purpose: Social-emotional skills training (SEST) programs are ubiquitous in American public schools. These programs are designed to promote social competence and to reduce children’s behavioral problems. However, studies of these programs have produced mixed findings. Indeed, discrepancies are often found among evaluations of the same program. A potential factor contributing to these discrepancies is varying implementation. Differential implementation often results in varying dosages of treatment (i.e., treatment exposure), and evaluating these dosage effects is a crucial element of intervention research. This study investigated dosage effects of the Making Choices (MC) program—a SEST program for elementary school children. Data are from a cluster randomized trial involving 10 schools randomly assigned to MC or a control condition. Because dose variations are not randomly assigned, assessing dosage requires controlling for selection effects. To control for selection bias, the dosage analysis employed a new method using Generalized Propensity Scores (GPS).   

 

Methods: The analysis focused on two cohorts of third-grade students who participated in a trial of the MC program. The final sample consisted of 400 students from 30 classrooms in 5 intervention schools.

Number of minutes of training was used as dosage measure. The designed length of MC training was 1120 minutes for third graders and 320 minutes for fourth graders. Reflecting implementation variation, the observed number of minutes ranged from 268 to 2,340 in Grade 3 and from 220 to 600 in Grade 4.

The dosage analysis used a GPS method that deals with continuous treatment measures. The GPS was estimated by modeling the conditional density of the treatment given covariates using a normal density function. The covariates included variables measured at individual, classroom, and school levels. Using the GPS method with minutes of SEST exposure, dosage effects of the MC program were evaluated at the end of Grade 3 and Grade 4 for eight key outcomes. To control for rater effects, change scores of the outcomes within Grade 3 and Grade 4 were used as the dependent variable.

Results: Findings indicate that the intervention had overall dosage effects at the end of Grade 3 on social competence (p = .001), emotional regulation (p =.040), and social aggression (p = .003). No significant dosage effects were found at follow up at the end of Grade 4.  For outcomes on which the intervention had significant overall dosage effects, average dosage effects were computed within 10 treatment levels of implementation—each included approximately 10% of the participants. Findings indicate a positive relationship between dose and both skill and behavioral outcomes. However, the relationship is not strictly linear.

Conclusion and Implications: At scale, the implementation of the MC curriculum by classroom teachers varied greatly and outcomes were related to this variation. Overall, positive findings emerged on relationship between SEST exposure and key intervention outcomes. Findings from dosage analysis provide insights into factors that have contributed to the mixed findings from evaluation studies of SEST program as well as crucial information for finding optimal dosages of treatment.