Competency Development for Leaders in Child Welfare

Purpose: The Children’s Bureau’s National Child Welfare Workforce Institute (NCWWI) Leadership Academy for Middle Managers (LAMM) provides leadership competency-based training through pre-work, 6 days of residential training, and post-training supports. This study describes the development and testing of a leadership competency self-assessment tool that measures learning and transfer of learning as it evolves from knowledge-based to skill mastery through application. A common problem of self-assessments is inadequate testing for validity and reliability (Hinkin, 1995). To effectively use competency self-assessments as a measure of learning and transfer of knowledge, they must be developed in alignment with a strong competency-based training and administered to a large enough sample to allow for standard content or construct validation procedures. The current study aimed to develop a valid and reliable competency assessment for the NCWWI LAMM that measured learning and continued skill development in the year following training.

Method: LAMM curriculum designers and evaluators collaborated to develop a 28-item self-assessment (5-point Likert-style scale) that reflects the knowledge and skill competencies addressed in the LAMM training. The assessment was administered to LAMM participants to capture transfer of learning from a knowledge and understanding level of competency at pre-post (N = 223) to the gradual adoption of behavior and performance level of competency at 3 (N = 180), 6 (N = 151), and 12 months (N= 133) post-training.

Results: Reliability analyses on the self-assessment revealed strong internal consistencies at each time point: Cronbach’s α = .91 (pre-training), .95 (post-training), .94 (three months), .94 (six months), and .95 (12 months). Exploratory Factor Analyses indicated two underlying factors; here we focus on the one measuring general leadership skills (Cronbach’s α = .88, .90, .89, .86, and .89, at each timepoint). Using repeated measures ANOVA to test for change over time, we found significant competency gains from pre- to post-training, F(1, 390) = 1480.86, p < .001, η² = .79; as well as significant gains from pre-training to the 12-month follow-up, F(4, 116) = 195.92, p < .001, η² = .62. Over time, average leadership skills spiked at post-training then decreased slightly at each subsequent measurement point.

Conclusions and Implications: Findings from this study confirm the utility of a self-assessment tool for measuring change in leadership skills and indicate positive growth in competency among LAMM participants. Gradual “skill decay” is a common problem with training programs (Arthur, Bennett, Stanush, & McNelly, 1998) and speaks to the need for continued implementation of transfer of learning supports for participants. Based on the learning theory that participants begin understanding key constructs during a training event but acquire skills through practice on the job in a supportive transfer climate (Holton, Bates, & Ruana, 2000), we developed the self-assessment to focus on the knowledge and understanding level at pre- and post-assessments, and on performance level at follow-up assessments.  Discussion will include challenges associated with analyzing repeated measures designs with a measure that changes at different time points.