Coping With Publication and Reporting Biases in Research Reviews

Readers should expect research reviews to provide comprehensive and accurate summaries of empirical evidence. However, most published reviews and meta-analyses are based on convenience samples of published studies; these reviews can lead to the wrong conclusions.

The best available evidence indicates that (a) fewer than half of all completed studies are published, (b) publication status is not a good proxy for methodological quality (many high-quality studies remain unpublished); and (c) the odds of publication are two to three times greater for studies that have statistically significant, “positive” results, compared with the odds of publication for equally-valid studies that have null or “negative” results (Dwan et al., 2008; Song et al., 2009, 2010). Reporting, publication, and dissemination biases are cumulative and this skews available research results.

For example, in the field of intervention research, it is important to know which interventions have equivalent effects and which may be harmful. However, studies that shed light on these two issues are systematically under-represented in the published literature. Reviews of published intervention studies tend to overestimate effect sizes (Altman, 2006). In some instances, positive effects are fully explained by reporting and publication biases (Kirkham et al., 2010). How are we to cope with this skew in the published literature? 

This workshop reviews empirical evidence on selective reporting, publication, and dissemination biases in the scholarly literature. Presenters discuss strategies for coping with these biases, particularly in systematic reviews of research. Prior knowledge of systematic review methods is not required. 

Presenters will describe

1. Empirical evidence on the extent of selective reporting, publication, and dissemination biases in the scholarly literature;
2. Strategies for limiting these biases in the scholarly literature;
3. Methods for limiting the impact of these biases in systematic reviews (gray literature search strategies); and
4. Strategies for assessing and adjusting for publication and reporting biases in reviews (limitations of failsafe-N; use of trim-and-fill analysis, contour-enhanced funnel plots, and regression models).

Pedagogical methods include lecture, discussion, software demonstration, and handouts.