Combining Practice Wisdom and Implementation Drivers to Develop a Measure of Data-Driven Decision Making

Background: Evidence-based early childhood programs have been found to facilitate positive child and family outcomes, including healthy parent-child relationships, stimulating and safe home environments, maternal-child health, and child safety (Barton, 2016; Boller et al, 2010; Sweet & Applebaum, 2004). Despite these positive impacts, broad access to quality early childhood programs has been hindered by government budget constraints (National Home Visiting Resource Coalition, 2017). While implementation of evidence-based programs is known to require ongoing use of data to improve practice and decisions and ensure successful implementation (Fixsen et al., 2019), many providers have not fully embraced continuous use of data to inform practice and policy (Coultan et al., 2015). Further, few practical measurement tools exist to support data-driven decision-making (DDDM). Using an implementation science framework and engagement with community providers, this study sought to document, develop, and test a tool to assess DDDM among early childhood programs.

Methods: Researchers developed a 54-item questionnaire around the nine Active Implementation Drivers (Fixsen et al., 2019) using a 5-point Likert scale. In addition to applying theoretical constructs of implementation drivers, community-based early childhood providers served on an expert panel for development and pilot-testing. The DDDM Questionnaire was piloted with 173 early childhood program administrators, representing a 32% response rate. Analyses followed Goodwin’s (2002) guidance by examining three of five categories of validity evidence, including (1) test content, (2) internal structure, and (3) relationships to other variables. The study involved specifying a nine-factor confirmatory factor analysis (CFA) to determine if validity evidence emerged in support of the nine implementation drivers as an underlying structure for examining DDDM.

Results: Expert panelists provided three rounds of feedback, which resulted in refined instructions and item additions and subtractions. Mean scores on the 54 DDDM-Questionnaire items ranged from 2.7 to 4.5. Among the nine implementation driver subscales, systems intervention was highest (M=4.2, SD=.52) and training was lowest (M=3.3, SD=.64). The CFA’s model fit indices indicated fit with the proposed 9-factor model (CFI=.98; TLI=.97; RMSEA=.03, 90% CI .021–.036). The ratio of the chi square statistic relative to the degrees of freedom (χ2/df=1.14, χ2(1341)=1534.65, p<.001) also suggested overall goodness-of-fit. Regarding internal consistency, 8 of 9 factors ranged from .73 to .90 and one factor (staff selection) was slightly below the threshold at .67. Further supporting these confirmatory analyses, the study found no statistically significant evidence to suggest DDDM is related to other external variables, including participant or program characteristics.

Conclusions: This study demonstrates an approach that honored providers' expertise as well as theoretical frameworks on implementation science. Given the encouraging results on model fit, future research could examine whether the nine implementation drivers are valid longitudinally and across different programs/contexts. For early childhood programs as well as other types of evidence-informed interventions and services, the DDDM-Questionnaire may have utility for assessing readiness for successful implementation that is informed by data across multiple implementation drivers. In sum, the DDDM-Questionnaire may help administrators understand their organizational readiness for DDDM and where they have strengths and areas for improvement.