Beyond Averages: A Flexible Distributional Approach to Evaluating Intervention Effectiveness in Middle Schools

Background and Purpose: In the wake of the initial COVID-19 impact, behavioral issues in U.S. schools have been on the rise (EdWeek Research Center, 2023). Efforts to identify evidence-based interventions (EBIs) are critical to ensuring students receive effective supports to address rising behavior concerns. However, substantial debate surrounds traditional methods used to evaluate intervention effectiveness in determining what constitutes an EBI (Wiedermann et al., 2022). Specifically, traditional mean analysis falls short in providing a comprehensive description of intervention impact, primarily due to its inability to account for other parameters influenced by the intervention. The purpose of this study is to address these shortcomings and present a more nuanced evaluation approach.

Methods: Unlike traditional mean analysis, the generalized additive models for location, scale, and shape (GAMLSS) offers a flexible framework that can handle a broader range of relationships when evaluating the impact of an intervention on study outcomes. Additionally, GAMLSS enables a comprehensive understanding of intervention effects by simultaneously modeling means, variances, skewnesses, and kurtoses. Given these advantages, the current study applied GAMLSS to a diverse sample of middle school students within a large-scale randomized controlled trial (i.e.,102 teachers and 1,450 6^th-8^th grade students) to evaluate the impacts of the CHAMPS classroom management program (Herman et al., 2020) on student outcomes.

Results: Inflated mixed probability distributions (i.e., inflated beta distribution) were modeled using GAMLSS to examine the overall treatment effects on student behavior. Our findings indicated that CHAMPS improved student concentration problems (OR = .84, 95% CI [-0.25, -0.06]), emotion regulation (OR = 1.09, 95% [-0.18, -0.01]), and disruptive behavior (OR = .55, [-0.23, -0.04]), with null effects observed for prosocial behavior.

Conclusions and Implications: The findings by Herman et al. (2020), which used the traditional analysis method, identified that CHAMPS improved teacher ratings of student concentration problems (d = -0.18), with null effects observed for disruptive behavior, emotion regulation, and prosocial behavior. However, this study employing the GAMLSS approach uncovered noteworthy insights. Our findings indicate significant improvements not only in concentration problems but also in prosocial behavior and disruptive behavior. This disparity highlights the enhanced sensitivity of GAMLSS in detecting intervention effects that might be overlooked when using traditional methods. In addition, the comparison underscores the potential limitations of traditional mean analysis in capturing the multifaceted nature of behavioral outcomes.

Beyond demonstrating positive effects of classroom behavior management on key student behavioral outcomes, the results offer the chance to capture additional aspects of the data distribution, including variance, skewness, and kurtosis. Moving forward, the adoption of distributional approaches, such as GAMLSS, proves invaluable in understanding the intervention effectives and guiding evidence-based practices in addressing the behavioral challenges prevalent in U.S. schools post COVID-19.