Saturday, 15 January 2005 - 4:00 PM

This presentation is part of: Methodological Issues in Randomized Trials

Efficacy versus Intent-to-treat Analyses: How Much Intervention Is Necessary for a “Desirable” Outcome? Should You Include in Analyses Participants Who Were Assigned to Treatment But Who Had No Exposure to It?

Purpose: The purpose of this presentation is to compare the efficacy subset (ES) and intent-to-treat (ITT) perspectives in intervention research. In ITT, all individuals consenting to participate in a study are included in outcome assessments. Latent in ITT is the argument that factors causing low service dosages in research may operate also when interventions are brought to scale. Thus ITT is thought to provide more ecologically valid estimates of outcomes. Still, it is incumbent on researchers to determine the minimum level of intervention required to achieve desirable outcomes. This argues for ES analyses, which estimate outcomes at different levels of exposure to interventions.

Methods: ES and ITT outcomes are estimated from a study of Making Choices, a prevention program delivered over three years to 3rd-grade students in two schools in a southern state. Third graders in 2000-01 (cohort 1; n=182) received a routine health curriculum. Third graders in 2001-02 (cohort 2; n=173) received the MC program delivered by intervention specialists. Third graders in 2002-03 (cohort 3; n=198) received the MC program plus a supplemental intervention designed to involve classroom teachers and parents in MC-related activities. The cohorts did not differ on pretest measures.

Hierarchical linear modeling (HLM) is employed to estimate outcomes across six pre-posttest, teacher ratings of children’s behavior – cognitive concentration, social contact, social competence, authority acceptance, relational aggression, and overt aggression. Using HLM, student-level models regress posttest scores from each measure on covariates - pretest scores, race/ethnicity, gender, school, and risk exposure. Then classroom-level models (j=29) regress intercept and, where fit improved, slope estimators from student-level models on intervention indicators.

Although 86% of the Cohorts 2-3 children received at least 85% of the MC protocol, variation in exposure (due to absences and conflicting class schedules) permitted comparison of the ITT and ES perspectives. For ITT, all children regardless of exposure were included. In sequential testing of subsamples with differing dose levels, ES outcomes were estimated from exposure to “at least one session” to exposure to every session.

Results: Using ITT, Cohort 2 students were less relationally aggressive, and they had greater social contact. Also less relationally aggressive, Cohort 3 students were rated higher on social competence and cognitive concentration. Using ES, efficacy improved across the 70-95% exposure intervals. At dosages of 85%+ for Cohort 3, authority acceptance and overt aggression rose to significance, presenting potentially important findings not observed in ITT. Students above/below the 85% threshold did not differ on pretest or demographic measures.

Implications: The results suggest that children receiving 85%+ of MC experience a broad range of desirable behavioral outcomes. Contrasting ITT, ES allows researchers to identify dosage thresholds at which different kinds of outcomes may be observed. Premature use of ITT may achieve the unintended consequence of declaring potentially effective programs to be ineffective. At the same time, failure to invoke ITT may result in an over-estimation of potential outcomes when programs are brought to scale. A range of dose-related analyses appear required to properly estimate treatment effects and establish empirically-based guidelines for program implementation.

See more of Methodological Issues in Randomized Trials
See more of Oral and Poster

See more of Celebrating a Decade of SSWR (January 13 - 16, 2005)