Giving Practitioners the Psychometrically Sound Measures They Need: Using Multilevel Techniques, Data from Multiple Sources, and Control Variables to Demonstrate the Reliability and Validity of a New School Climate Measure

Background: NASW standards for school social work encourage data-informed school climate interventions. To intervene effectively, practitioners need psychometrically sound measures. Although existing measures often aggregate student perceptions to the school-level, multilevel techniques have not been used to examine the reliability of any existing measure. Additionally, extant validity studies could be biased by shared-method variance and omitted variables, as most rely on data collected from a single source and fail to account for the strong correlation between school climate and school demographic composition. The present study demonstrates how social work researchers might address these issues while examining the psychometric properties of a new school climate measure.

Methods: Data included administrative records and student and teacher surveys collected over three years in a medium-sized, U.S. public school district (N=17,971-18,009 students, 1,364-1,597 teachers, in 31-37 schools). Measures included student perceptions of school climate, administrative data on suspension and teacher turnover rates, as well as teacher perceptions of: teaching and learning conditions, community support for learning, and student conduct. Race/ethnicity, socioeconomic status, and prior achievement were key control variables and included in a school-level advantage index. School-level school climate reliabilities were calculated following established multilevel methods (Barnett, Marshall, Raudenbush, & Brennan, 1993) on raw student responses as well as the residual of an Ordinary Least Squares regression predicting student responses by key controls. Validity analyses used generalized estimating equations (McNeish, 2014) to examine the correlation between aggregate perceptions of school climate in one year and each administrative and teacher perception outcome the following year with and without controls for school advantage.  

Results: School-level reliabilities were quite high but dropped slightly when race/ethnicity, socioeconomic status, and prior achievement were controlled, Year 1: ω=.85 vs. ω=.79, Year 2: ω=.91 vs. ω=.85, Year 3: ω=.86 vs. ω=.81. A standard deviation increase in school climate in one year was associated respectively with a 54% and 51% decrease in school suspension and teacher turnover rates the following year. Holding school composition constant, the decrease shrunk to 41% and 24% respectively. Similarly, a standard deviation increase in school climate in one year was associated respectively with a .15, .47, and .70 standard deviation increase in teacher perceptions of teaching and learning conditions, community support for learning, and student conduct the following year. In models controlling for school composition, the association was .20, .81, and .57 standard deviations. All relationships were significant (p≤.05) except the relationship between school climate and teacher turnover rates in models controlling for school composition (p≤.10).

Implications: This study demonstrates how multilevel reliability techniques, data from multiple sources, and controls for school demographic composition can be used to address key methodological limitations of extant school climate research. While more studies using similar methods are needed, the findings of this study may help practitioners and scholars estimate the direction and extent of bias caused by the widespread omission of student and school advantage in previous research. Findings also indicate that practitioners can use measure examined in this study to reliably and validly measure school climate.