The Role of Adherence, Dosage, and Participant Responsiveness in Classroom Teacher's Assessment of Overall Implementation Strength

Background and Purpose: Social emotional learning (SEL) interventions in schools are intended to nurture social and emotional capacities in children, disrupt trajectories toward problem occurrence, and strengthen prospects for school and life success. SEL interventions work by increasing social and emotional skills; improving student attitudes about themselves, others, and school; enhancing social and classroom behavior; reducing emotional distress related to stress and depression; and promoting academic achievement (Durlak & DuPre, 2008). Although SEL interventions have been tested and demonstrated to be effective for supporting positive youth development, there is an underdeveloped science to inform their widespread delivery (Wandersman et al., 2008).

The Toolbox Project is an innovative, research-based, community-tested SEL curriculum for grades K-6. The program guides school and family support for their children’s social and emotional development through the explicit cultivation of students’ innate strengths. Toolbox has been implemented in over 100 schools in over 30 cities. While some initial evaluation of the Toolbox Project has focused on process outcomes as well as preliminary student, family, and institutional outcomes, an exploration into the strength of program implementation that is required to achieve these outcomes has not occurred. This study seeks to understand implementation characteristics that relate to overall implementation strength.

Methods: Data are from a district-wide, online survey administered within 11 Berkeley Unified School District elementary schools where Toolbox was implemented during the 2014-2015 year.  The survey of district staff yielded an overall 73% response rate. Multivariate regression was used to estimate the effects of specific aspects of implementation (e.g., adherence, dosage, participant responsiveness) on overall implementation strength, as reported by teachers. Adherence was measured by asking teachers how closely they followed the approach outlined in the curriculum guide while teaching a sample lesson. Dosage was measured by the number of lessons taught from the Toolbox Curriculum Guide during the academic year from a total of 14 lessons (grades K-3) or 16 lessons (grades 4-6). Participant responsiveness was assessed through teacher ratings of student engagement on a sample lesson.

Results:    Teacher assessment of their own implementation strength ranged from 1 (low) to 10 (M=6.02, SD=2.06). Adherence was moderate (1-4 scale; M=2.5, SD=6.8), meaning that teachers typically made more than minor changes to the lessons.  Dosage was also moderate for both grade levels; grades K-3 (M=8.79 of 14 lessons, SD=3.6) and grades 4-6 (M=8.38 of 16 lessons, SD=3.3). Participant responsiveness was high (1-5 scale; M=4.03, SD=.81).  Results indicated that teacher adherence and teacher dosage, but not student responsiveness, significantly predicted teacher assessment of the strength of their implementation. Dosage explained a much larger amount (34% for K-3, 28% for 4-6) of the variance in implementation strength than adherence (4%).

Conclusions and Implications: Findings suggest that adherence and dosage contribute to teacher assessment of the strength of their program delivery.  Teachers seem to appreciate dosage more than adherence as essential to implementation success. Implications for implementation monitoring and strengthening will be discussed.