Treatment Fidelity In the CareerStart Random Controlled Trial: A Mixed Methods Implementation Study

Background and Purpose: Given current emphases on standards-based accountability and research-based interventions, implementation fidelity has become a matter of great interest to educational policymakers and researchers. Clearly, rigorous implementation of interventions requires uniform, consistent application of the treatment. Still, O'Donnell (2008), in her extensive review of fidelity in educational research, distinguishes between fidelity and the adaptation of interventions during the implementation process, which she speculates may be “inevitable and perhaps itself related to program success” (p. 53). This paper explores this assertion by employing mixed methods research on a rigorous evaluation of CareerStart. We identify and consider the impact of differences in levels of program implementation fidelity.

Methods: This study draws on qualitative and quantitative data collected at 7 treatment middle schools in a 14 school random clinical trial implementing CareerStart over a three-year period. First, using field staff reports we identified schools as low or high fidelity. We then examined quantitative data to provide independent validation of these findings, using teachers' reports of number of CareerStart lessons taught. We compared school-level means of teacher reported lessons and estimated multilevel models nesting teachers in their schools with teacher reported lessons as the dependent variable. Each school's random effect then represented deviation from the grand mean of all schools. We also compared high and low fidelity schools, using t-tests to compare student demographic and achievement data, and teacher survey responses regarding their implementation of and attitudes towards CareerStart. Finally, we analyzed focus group data to identify themes in student and teacher experiences of CareerStart implementation. We compared these themes across schools in order to indentify similarities and differences between low and high fidelity schools.

Results: Quantitative results supported field staff reports on fidelity to CareerStart's design, with three schools identified as low implementation schools. These schools had a number of achievement indicators that were significantly lower than those at higher fidelity schools, and also had twice as many free/reduced-price lunch eligible students (92% compared to 46%) and over three times as many Latino students (41% compared to 12%). Focus group data analysis revealed that high fidelity schools had a more robust CareerStart implementation, but that this frequently involved teacher adaptation of CareerStart materials when found insufficient. Teachers at low fidelity schools also reported difficulties, but instead of adapting available curricula simply did not deliver the lessons. Students at high fidelity schools more readily identified examples of career-relevant instruction in their classrooms, described these examples in greater detail, and reported career-relevant learning across a wider range of subjects and teachers.

Conclusions and Implications: This study demonstrates how intervention fidelity lends itself to a mixed methods approach and involves both adherence to intervention plans and adaptation to setting demands. Our findings support the idea that adaptation may be a necessary component of interventions in the interest of fidelity. As such, in the case of CareerStart, curricular materials that support implementation fidelity may need to include tools and relevant content that teachers can adapt as fits their particular skill sets, classroom context, or students.