Survey Redesign to Monitor Progress Towards Eliminating the Opportunity Gap: A Research Practice Partnership with Seattle Public Schools

Background: Many school-based social emotional learning (SEL) interventions and school climate reform efforts have demonstrated efficacy in preventing behavioral health problems and improving academic achievement in schools (Durlak et al., 2011). However, disconnects between research priorities and needs of the local context often hamper schools districts’ ability to use existing survey tools to drive change efforts or evaluate progress. In partnership with Seattle Public Schools (SPS)—a large, urban school district with a highly diverse student body— this project uses principles from research-practice partnerships (Coburn et al., 2010), to redevelop the SPS student school climate and social emotional competence survey, integrating analyses of past year’s survey and the district’s goal to eliminate the opportunity gap. The current annual school climate survey was analyzed for its alignment with school district goals and practices, its ability to facilitate and monitor school improvement, and for its psychometric properties. A particular focus was the relevance of the survey across racial and language groups, since a focus of the school district is to use survey results as part of an effort to eliminate the opportunity gap for students of color.

 Methods: All students (N=29,593) from the 94 SPS schools in grades 3-12 take the annual school climate survey (44.4% male, 45.8% female, 6.5% prefer not to state; 59.2% English at home). Students self-reported their race: 15.1% Asian, 10.7% African American, 6.7% Latino or Hispanic, 14.9% Multiracial, 1.5% Native American, 1.7% Pacific Islander, and 44.9% White. Confirmatory factor analysis (CFA) was used to make initial refinements of the measures.  Multiple group CFA models were then used to assess configural, metric, and scalar invariance across gender, language, and racial groups. Results are being used to redevelop the survey to better align with district goals.

 Results: The CFA resulted in a five-factor model for climate and a two-factor model for social emotional skills fit the data well (RMSEA=0 .04; CFI=0.92; TLI=0.91), with sufficient factor loadings (0.50-.79) and internal reliability (Cronbach’s Alpha 0.72-0.85).  The correlation between high-order global factors for climate and social emotional skills was r=0.61 (p<.001).  Multiple group invariance for differences in measurement across racial/ethnic groups supported configural and metric invariance. Fit was substantially worse, however, for models testing scalar invariance model that included equality constraints on intercepts of indicators (RMSEA=0.04, CFI=0.91; TLI=0.91; Cheung & Rensvold, 2002), suggesting that some items differed across racial/ethnic groups in terms of their “severity” with respect to underlying constructs. These results are guiding the reformulation of the constructs of interest to the school district, to align to the local context and practice goals of schools. 

 Conclusions: Partnering with SPS resulted in a psychometrically stronger, practice relevant survey, making it more useful to schools as a tool for monitoring progress. Results suggested important areas not captured by the previous survey, such as the importance of identity safety and adult-student relationships especially for historically underserved students of color. Measuring constructs related to equity, and ensuring the survey is relevant to all student groups are first steps towards eliminating the opportunity gap.