Advances in Single-Subject Design Effect Size for Meta-Analysis: Introducing a 4-Parameter Regression Model

Quantitative meta-analysis of intervention/service research has become an increasingly important tool for social scientists to objectively evaluate empirical evidence of psychosocial interventions so that social workers can effectively engage in evidence based practice. Unlike a plethora of meta-analyses of randomized-controlled-trials (RCTs), single-subject design (SSD) studies are often excluded from meta-analyses due to a lack of appropriate statistical methods for calculating effect size estimates (Shadish, Rindskopf, & Hedges, 2008). The exclusion of SSD studies has implications for practice evaluation and services research in social work because a substantial body of literature using SSD is in applied settings such as schools, social service agencies, and healthcare settings.

Traditional procedures, including visual analysis (Kratocwill et al., 2010), pre-post-score standardized mean difference (Busk & Serlin, 1992), and non-parametric percentage of non-overlapping data (PND) (Scruggs et al., 1987), have been increasingly critiqued for their low validity of statistical test, insufficient treatment to the auto-correlated data structure, and, most importantly, for ignoring time trend as a confounding factor (Beretvas & Chungm, 2008). These critiques along with the imperative for SSD researchers to join the evidence-based-practice movement as equal participants have (re)directed methodological research in SSD effect size and meta-analysis to regression approaches (Beretvas & Chung, 2008; Maggin et al., 2011).

Huitema and McKean (2000) suggested a 4-parameter regression approach, an extension of the piecewise regression suggested by Gorman and Allison (1996), that allows appropriate parameterization of two coefficients to describe change in intercept and in slope from phase to phase in SSD studies. According to latest simulation studies, this 4-parameter regression approach not only addresses time trend as a confounding factor but also improves the validity of statistical test (Moeyaert et al., 2014; Ugille et al., 2012). Most importantly, when used in conjunction with multi-level modeling, the 4-parameter approach facilitates a more principled way of conducting meta-analysis: a three-level meta-analysis (Heyvaert et al., 2012). The three-level meta-analysis partitions variability in SSD effect sizes into the measurement sampling variations (across time) for each individual (level 1), variation across multiple effect-sizes/individuals within a study (level 2), and variation across studies (level 3).

Workshop presenters will explain the need for and demonstrate the use of the 4-parameter regression approach for calculating effect size estimates from SSD studies and and procedures for conducting multi-level meta-analysis of SSD studies. Material will be provided that will include multiple examples of how to calculate SSD effect size estimates and ways to conduct multi-level meta-analysis. This 90-minute workshop is designed to help participants: 1. Understand the issues, challenges and complications of ignoring SSD data structure using existing methods; 2. Understand the 4-parameter model and the interpretation of the four regression coefficients; 3. Learn how to set up SSD data in excel and implement the 4-parameter model and multi-level meta-analysis in R; 4. Learn how to interpret the effect sizes calculated using the 4-parameter model and examples of how to interpret the overall results of the multi-level meta-analysis of SSD studies.