Utilizing 360-Degree Assessments for Program Evaluation and Leadership Development with Child Welfare Managers

Background and Purpose:  Organizations often implement 360^◦ assessments to improve leadership and organizational performance. Research shows that for managers, 360^◦ feedback promotes self-awareness regarding skill strengths and need areas. The current study utilized 360^◦ assessments to enhance child welfare middle managers’ self-awareness as they progress through a leadership development program. This paper presents information on the development and baseline testing of a 360^◦ assessment of leadership competencies for effective management in child welfare and other human services agencies.

Method:  Evaluators developed leadership competency items based on the program’s leadership development framework that includes five domains: Leading Change (e.g., Encourages new ideas and innovations), Leading in Context (e.g., Builds consensus with partners by considering input from various parties), Leading People (e.g., Fosters an inclusive environment that values all types of diversity and opinion), Leading for Results (e.g., Sets priorities and determines resources requirements), and Fundamentals (e.g., Sets a personal example of what he/she expects from others). The leadership domains are addressed during a 3-day residential training for managers who also complete pre-workshop activities online and participate in pre-post coaching sessions.  Using Qualtrics 360 Software, the 34-item leadership competencies scale (comprised of three to ten items per domain) was administered online to 33 child welfare middle managers in two states. Managers also nominated up to nine others from their workplace (staff, peers, and supervisor) to rate their leadership behaviors. The leadership competencies scale demonstrated adequate reliability (whole scale Cronbach’s alpha = .96 with alphas ranging from .85 to .92 for domains). For quantitative analyses, reports from others were aggregated by 360 participant. Respondents also provided open-ended feedback to prompts asking about participants’ leadership strengths and need areas, which helped contextualize findings.

Results:  Intraclass correlations (ICC) that were calculated to compare self-assessments with raters’ assessments on overall leadership and each domain revealed very low agreement (high concordance is typically > .70, but in this sample, ICCs were less than .20). To determine whether self-report differed significantly from other-report of leadership behaviors, we conducted paired samples t-tests and found significant differences on Leading Change, t(32)=3.52, p=.001; Leading in Context, t(32)=3.72, p=.001; Leading People, t(32)=2.08, p=.05; Leading for Results, t(32)=2.34, p=.02; and Overall Leadership, t(32)=3.27, p=.003. Self-assessments did not differ from others’ assessments on Fundamentals, t(32)=0.59, p=.56. In all domains, others’ ratings were higher, on average, than were self-ratings.

Conclusions and Implications:  Discrepancies between self-reports and other-reports can indicate lack of self-awareness, inability of reporters to actually observe certain competencies, or item ambiguity that makes an item difficult to endorse, all of which may account for the significant differences found between self- and other-reports in this study. The lack of difference on Fundamentals may indicate that behaviors in that domain are easier for others to observe and rate. Discussion will address the importance of using a framework for tool development, challenges in selecting response options that will provide variability, and implementation considerations such as how to summarize and utilize findings, and strategies for using 360s to support leadership growth such as coaching and ongoing support.