Setting up for Success: Principles of Big Data Practice and the Science of Implementation

Current thinking on implementation science recognizes the importance of a distinct group of pre-implementation activities to the success of the broader implementation effort. Drawing on that literature, this workshop reconsiders the role that administrative data systems (“big data”) can play in maximizing the potential for implementation success, particularly at the pre-implementation or exploration stage.  Specifically, the panelists will focus on four principles of big data practice that, if adhered to, can aid in intervention, implementation, and evaluation design: (1) start the process by asking a question, (2) take a disciplined approach to converting data to evidence, (3) commit to the cyclical process of improvement using evidence, and (4) arrange and analyze the data in ways that maximize knowledge development.

The first panelist will frame the workshop in terms of the first principle – the need to initialize any intervention planning by asking specific, answerable questions. For illustrative purposes, the focus is on targeting, specifically, who to target (triage), when to intervene (timing), how to intervene (the intervention), and how much (dosage) of the intervention or service to provide.

The second panelist will discuss intervention timing by way of the second principle. Timing the delivery of an intervention requires disciplined consideration of the risk group (denominator) and the window during which change is expected to occur. Targeting through the use of administrative data can be used during an intervention’s design phase to decide when the treatment should begin in order to optimize the chance that the intervention has its intended effect.

The third panelist will discuss the third principle, focusing on dosage in the context of intent-to-treat evaluation frameworks. In intent-to-treat designs, children eligible to receive the treatment but who did not receive the treatment (or did not receive the full amount of the treatment) are included in the analysis of program effects. Because the untreated group is often larger than the treated group, effects can be difficult to detect. Thinking about the issue of dosage in advance of implementation can help planners make strategic decisions about (1) how much of an intervention is needed to see effects and (2) how to target resources so as to improve the likelihood of achieving dosage goals. This could amount to rethinking the target population, or to making rapid implementation adjustments.

The last panelist will discuss the fourth principle using examples of triage – the process of not only locating those who need the intervention, but also those who will benefit from it. Arranged and analyzed appropriately, administrative data can be used to identify service or risk histories so that an intervention can be directed to individuals who need the service or intervention but who also stand the best chance of benefiting from it.

Panel members will briefly outline each of the principles for the first hour, providing examples from applied research. They will leave at least 30 minutes for questions, soliciting examples from audience members to discuss ways that the principles may be applied to current and upcoming intervention designs and evaluations.