Beta Test and Qualitative Feedback of Newly Translated Smarts Middle School Curriculum

Background and Purpose: This paper describes how the Self-Monitoring and Regulation Training Strategy (SMARTS) school-based intervention was systematically adapted and pilot tested to be developmentally appropriate from elementary to middle school students using research-based translational steps. SMARTS is a school-based intervention designed to enhance students’ skills to improve self-efficacy and confidence in their ability to self-monitor personal goals and regulate behaviors in an academic setting.

This study occurred within the context of a four-year quasi-experimental research study, funded by the United States Department of Education. The overall purpose of the current study was threefold: (1) highlight the SMARTS middle school intervention developed using participatory action research (PAR) and coproduction (CP) methods; (2) describe the step-wise approach used to translate the curriculum; and (3) explore quantitative (e.g., fidelity) and qualitative (e.g., focus group) data to examine the implementation and use of the SMARTS during a pilot test.

Methods: The present study employed a mixed method approach utilizing the following data sources: direct supervision and focus groups, pre- and post-test surveys, and feasibility surveys. Direct supervision included student participation in a 10-lesson intervention with two lessons each week, and weekly processing meetings. Participants (N=27) included a mix of students (n=7), teachers (n=14), and school support specialists (n=2) at local middle schools (grades 6^th to 8^th). Graduate students (n=4) from a local university were recruited to facilitate groups.

Quantitative data was analyzed, utilizing paired t-tests and the Reliable Change Index to test differences between pre- and posttest surveys for each student participant. Additional quantitative analyzes included mean and standard deviations for demographic information and feasibility surveys. Qualitative components of the study included a 1-hour semi-structured focus group with participants to explore their general reflections on SMARTS intervention materials, facilitation, and participation later analyzed using content analysis methods.

Results: Results of this beta test indicated SMARTS was well accepted with good feasibility reported by students, teachers, and school support personnel. Despite an underpowered sample size, results indicate a clinically significant impact on student participants in SMARTS. More than half of students (N=4) showed clinical improvement in interpersonal relationships (e.g., improvement in friendship interactions) and improvement in reducing externalizing behaviors.

Focus group data indicated students reported a positive preference for the lessons, autonomy setting their own goals, and the positive behavior program (e.g., SMARTS Store). Teachers and school support professionals shared the intervention was accessible and feasible to incorporate in their daily schedules. Some teachers noted observable behavior changes in their students (e.g., motivation and engagement during class). Additionally, teachers provided suggestions for future improvements to the SMARTS intervention (i.e. website component).

Conclusions and Implications: The present study served as a beta test to investigate preliminary data and test the feasibility of the SMARTS intervention for middle schoolers prior to the multiple year quasi-experiment. This study allowed investigators to note areas of acceptability and suggested improvement leading into the larger study.