Overviews of Reviews: Unique Challenges and Opportunities of Synthesizing the Syntheses

Background: Research synthesis, a rigorous approach to cumulate evidence, has become an important technique to manage, integrate, and summarize the rapidly expanding research literature. The number of research syntheses being published in the social sciences is also rapidly expanding, nearly doubling each year. Largely as a result of the increase in syntheses, including systematic reviews, researchers have begun to synthesize the syntheses. These syntheses of syntheses, otherwise known as overviews, have become an increasingly prevalent approach to synthesizing research. Although overviews are becoming prevalent, and may offer advantages over traditional research syntheses, overviews are a relatively nascent and undeveloped synthesis method that pose unique methodological challenges and may be problematic. While significant empirical work has been undertaken to inform and improve research synthesis methods to increase their credibility and validity, limited research or guidance is available to the overview author. The purpose of this study was to examine the methodological quality and current state of overviews, to provide further guidance for conducting overviews, and advance the evolution of overview methods. 

Methods: We conducted a systematic search for overviews aimed to synthesize more than one empirical education related review. One author searched multiple online databases and grey literature repositories. Two authors independently screened and selected studies and extracted data from included studies using a standardized codebook. At each stage, authors met to review inclusion decisions and coding. All discrepancies were resolved through consensus. The studies were analyzed descriptively; we aimed to elucidate all aspects of the overviews and calculate the proportion of characteristics reported across 17 different methodological characteristics. We also compared methodological quality and reporting of recent overviews to early overviews using t-tests. 

Results: Twenty-five overviews published between 1983 and 2012 met eligibility criteria for this study. Our analysis revealed that many commonly reported aspects of systematic reviews, such as the search, screening, and coding procedures, were regularly unreported. Of the 25 overviews included, 11 used a narrative synthesis technique (44%) whereas 14 used a quantitative analytic technique (56%): four elected to average the review results using a simple non-weighted average (16%), five chose to weight the results by the number of included reviews (20%), and five did not report how they synthesized results (20%). Overview authors rarely took into account overlap between reviews or up-to-datedness of reviews.

Discussion: The overviews offers an exciting, yet challenging new method for synthesizing and managing the ever expanding volume of research. Findings suggest, however, a serious lack of methodological reporting and use of rigorous methods for conducting overviews. Given the issues and limitations identified in this study, care must be taken in interpreting and using extant overviews. Moreover, to ensure the validity and utility of overviews to inform practice and policy, it is important that the conduct and reporting of overviews improve. As a result of this study, we developed conduct and reporting guidelines for overviews which will be presented and discussed.