Social Development Education: Developing and Testing the Making Choices Program with Propensity Score Matching

Background and Purpose: Social development programs are widely implemented in public schools. These programs are designed to reduce classroom disruptive behavior and promote academic achievement by strengthening the social skills of children. Although evaluations of social development programs have produced mixed findings, the Making Choices (MC) program has shown promise. It strengthens the social information processing skills of children, and control-group studies suggest that it reduces aggressive behavior while contributing to academic achievement. The purpose of this presentation is to describe findings from a cluster randomized trial of MC.

Methods: A total of 10 rural elementary schools were randomized to experimental or control conditions in a southern state. Teachers in the MC condition were trained to provide the MC program. In addition, they received training in attending to the social dynamics in their classrooms and in managing classroom behavior. From third through fifth grade, children in experimental schools received MC, with the bulk of training (M=17.5 hours) taking place in the third grade. “Booster” training of 8 hours was provided in each of the fourth and fifth grades. Teacher reports of child behavior were collected in the fall and spring of each year. In addition, teachers reported minutes of classroom time spent on MC content. To control for selection, attrition, and differential implementation, the evaluation employed a variety of propensity score models, including optimal full matching, matching estimators, and propensity score weighting in conjunction with hierarchical linear modeling.

Results: By the fourth and fifth grades, small positive behavioral and academic main effects emerged. Effect sizes were comparable to effects observed in routine educational studies of classroom content for such subjects as language arts. In addition to main effects, the findings from a dosage or efficacy subset analysis (ESA) with matching estimators – a procedure designed to control for selection in efficacy subsets – showed that booster dose (high, adequate, or none) had a significant impact, with comparatively greater declines in verbal and relational aggression for the fourth graders who received at least adequate exposure to MC (i.e., students who received 240 to 379 minutes of MC). High dose/exposure (i.e., students who received 380 or more minutes of MC) produced still larger declines relative to the zero-dose condition. In addition, the sample treatment effects for the treated group were significant for increases in social competence for the adequate exposure group. The ESA suggested that booster exposure of 4-6 hours produces positive effects. At that dose level, the MC program appears to promote prosocial behavior and reduce aggressive behavior. Booster exposure at a higher level may have some gain for 4th grade children, but it appears to have a negligible effect for 5th grade children.

Conclusions and Implications: Overall, the findings suggest that the intervention was effective and that relatively brief follow-up training has a significant effect on the classroom behavior of children. Selection, attrition, and differential implementation are common problems in evaluating school-based interventions. New methods using propensity scores provide innovative ways to deal analytically with these common design challenges.