Abstract: Predictors of Rater Bias in the Assessment of Social Emotional Competence (Society for Social Work and Research 21st Annual Conference - Ensure Healthy Development for all Youth)

Schedule:

Sunday, January 15, 2017: 10:45 AM

Balconies J (New Orleans Marriott)

* noted as presenting author

B. K. Elizabeth Kim, PhD, Assistant Professor, University of Southern California, Los Angeles, CA

Valerie Shapiro, PhD, Assistant Professor, University of California, Berkeley, Berkeley, CA

Sarah Accomazzo, PhD, Postdoctoral Scholar, University of California, Berkeley, Berkeley, CA

Joseph Roscoe, MSW, Doctoral Student, University of California, Berkeley, El Cerrito, CA

Background and Purpose: The Devereux Student Strengths Assessment Mini (DESSA-Mini; LeBuffe, Shapiro, & Naglieri, 2011) was designed to overcome obstacles to monitoring the growth of Social Emotional Competence (SEC) in the routine implementation of Social Emotional Learning (SEL) programs (Maras, 2015). This strength-based assessment system, which includes four interchangeable 8-item forms and a longer 72-item assessment, is now being used to electronically assess approximately 175,000 children each year in the United States.

Behavior rating scales like the DESSA-Mini strive to measure the frequency of behaviors that indicate age and context-appropriate emotional recognition and management, social awareness and interactions, and responsible decision-making. Behavior rating scales can be used repeatedly, across settings, and with multiple informants to capture an understanding of the child’s behavior over time and relative to a standardized reference group. Yet, behavioral rating scales have been criticized for their potential to incorporate rater bias into assessment scores (Elliot, Frey, & Davies, 2015).

Rater bias is “the presence of substantial and systematic error in ratings of performance or behavior caused by rater attitudes, beliefs, or experiences” (Mason, Gunersel, & Ney, 2014). Previous studies have considered sources of divergence between teacher ratings and a “true” criterion (Podsakoff, MacKenzie, & Podsakoff, 2012). Since SEC does not have a consensus indicator of “true” levels, this paper seeks to understand the variance in DESSA-Mini scores that is accounted for by rater characteristics.

Methods: Data were collected within a diverse California school district (13% Asian, 12% Black, 7% Filipino, 54% Hispanic, 9% White, 4% Other/Multiple/Not Reported). Rater characteristics were collected from (n=63) elementary school classroom teachers. These teachers assessed 1,676 students using the DESSA-Mini. The students were assessed to have similar levels of SEC (M=50.9; SD=11.7) as the national reference group (M=50.00; SD=10.00). Using an unconditional multilevel model, only 16% of the variance in student scores was attributable to characteristics of the teacher raters.

Findings: Teachers were 97% female, 60% White, 98% credentialed, and 16% district residents. Prior to training for the implementation of a new SEL program, 60% of teachers expressed that social emotional competence was “essential” (4 on 0-4 scale) to school success and 77% expressed typically feeling “eager” or “very eager” (3 or 4 on 0-4 scale) to adopt new initiatives. The extent to which teachers, at the end of an in-service training for the implementation of a new SEL curriculum, believed that the SEL program would benefit their students (B=1.90, p<0.10), intended to implement the program fully (B=1.00, p<0.05,), and did not anticipate implementation challenges (B=-2.46, p<0.01) each reliably accounted for shared variance in student assessment scores.

Conclusion and Implications: Identifying these sources of “rater bias” may lead to a) controlling for these sources of variance in research and b) shrinking these sources of variance through high-quality program selection, training, and implementation planning.