Collaboratively Developing and Validating Measures to Reflect Intervention Innovations When Exploring Provider Attitudes, Beliefs, and Practice Behaviors

Background: The Quality Improvement Center on Domestic Violence in Child Welfare (QIC-DVCW) aimed to develop, implement, and evaluate an innovative practice model to improve the lives of families experiencing DV and involved in the CW system. One component of this large project evaluated how to measure changes in attitudes and beliefs held by child welfare supervisors, child welfare workers, and community partners over time in addition to exploring how these were related to actual practice behaviors with survivors and persons who use violence. The process of how the team constructed meaningful measures of provider attitudes, beliefs, and practice behaviors relevant to the tested intervention approach evolved over time, shifting quickly and with some hiccups from a researcher-centered process to a partner-centered process to distribution of power across researchers and partners with varied practice and research experiences. This presentation highlights the collaborative process of scale construction and use of advanced quantitative statistical methods to highlight positional knowledge that increases relevance of the work while maximizing the technical ability to determine treatment effects.

Methods: Upon initial exploration at the onset of the project, previously validated measures did not align with partners’ stance or accurately reflect innovations intended by the current demonstration project. As a result, the research team engaged partners to co-develop their own scale that identified attitudes, beliefs, and behaviors aligning with the intervention approach. These items were collected using online survey techniques with a final analytic sample of N = 590 providers. The study team conducted confirmatory factor analysis (CFA) to assess psychometric properties of the items.

Results: Preliminary results from the measurement model suggested evidence of reliability and validity for the measure’s underlying factor structure. Thoughtful definition of priori decisions allowed us to easily identify constructs reflective of a survivor-centered lens in addition to potential residual covariance across items due to unintended imprecision of language. The measurement model including self-reported attitudes about work, beliefs about preparation to work, and actual practice behaviors demonstrated acceptable fits (CFIs = .91; TLIs =.90; RMSEAs = .06 [90% CI (0.05,0.06)]; SRMRs = .06). Longitudinal measurement invariance testing was conducted across two data collection time periods to evaluate stability of constructs over time.

Conclusions: Organizational and community partners have meaningful critiques of common approaches and measures traditionally used by researchers. Decentering expertise from the research/evaluation team was essential to define constructs and ask questions in our measures that felt ecologically valid to the context/situation and meaningful to those who were most likely to use the information. That being said, the research team have abilities to apply complex tools, such as design and statistical methods, that help to achieve the goal of measuring real differences across groups that could be attributed to an intervention. This approach distributes power and knowledge across the full researcher-practitioner team, balancing research training and knowledge with community and partners lived experiences of the phenomena being studied. The study team will share strategies to establish and maintain this balance throughout the study process and doing so within time constraints.