An Alternative to the Traditional Pretest-Posttest Design: Using the Retrospective Pretest-Posttest Design in Social Work Science

The purpose of this workshop is to share how researchers can use a retrospective pretest-posttest design to inform social work policy and practice.

The retrospective pretest-posttest design is used to examine the effectiveness of policies and practices. However, unlike the traditional pretest approach that gathers information before an intervention is implemented, a retrospective pretest gathers information only after the intervention has been completed. For example, after an intervention has been implemented, participants could be provided the following item: Please rate your confidence level with conducting research both BEFORE and AFTER participating in the training, using a scale of 1-5, with 1 being low confidence and 5 being high confidence. Then, participants rate themselves before (retrospective pretest) the program, and after (posttest) the program. This approach allows for data collection at one timepoint.

Though rarely used in social work research, this approach has been used for over 50 years in evaluation efforts because of its benefits over a traditional pretest-posttest design. It is cost-effective and convenient. It is useful when traditional pretests are not practical or feasible, such as when a policy has already been implemented. It is also beneficial when researchers are concerned that participants may inflate their knowledge (i.e., the Dunning-Kruger effect occurs) before beginning a program, such as trainings on cultural humility.

Despite the strengths of the retrospective pretest-posttest design, it has been criticized. It is subject to biases, including recall bias and social desirability. Further, it does not include a control group, it has limited generalizability, and it cannot be used to establish causality.

To address these criticisms, researchers have posited novel approaches that can increase the utility and effectiveness of the retrospective pretest-posttest design. Specifically, scholars have suggested using mixed methods, being intentional in selecting how response options are displayed, and including questions that assess whether changes are related to the program. These important advancements are essential for social workers to incorporate into their research, as they hold an important place in driving transformational change and improving policies and practices.

This workshop is for researchers new to the retrospective pretest-posttest design or individuals seeking to improve their use of it. Doctoral students are welcome. No experience with statistics or software packages are needed.

By the end of this workshop, participants will have answers to the following questions: (1) What is a retrospective pretest-posttest design? (2) When should and shouldn’t I use it? (3) What are the most common challenges to this approach, and how can I overcome these issues? (4) What statistical models can I run to analyze the findings? (5) How do I integrate mixed methods into this design? (6) How can I use this design in community-based research? (7) How can I use my results to inform social work policy and practice?