Friday, 14 January 2005 - 8:00 AM

This presentation is part of: Methods of Research Synthesis: Providing Credible Evidence for Practice and Policy

Assessment of Study Quality

Reviewers often assess the methodological quality of primary outcome studies in order to determine whether studies meet inclusion criteria and whether and how to take variations in study quality into account in the research synthesis (e.g., in sensitivity analysis or moderator analysis). The methods used to judge study quality can dramatically affect the results of a systematic review (Juni et al., 1999). There is general consensus among meta-analysts that unidimensional scales are inadequate to capture variations in study design and implementation, yet there is less agreement on which methodological features matter most (Valentine, 2004). More than 50 study quality assessment scales have been developed, but there is little information on their reliability (Maher et al., 2003).

Purpose: The purposes of this paper are to describe the most prominent and divergent methods of quality assessment and to illustrate, through their application to the same set of studies, how they point to differing conclusions regarding study quality.

Methods: To achieve these purposes, the characteristics of four study-quality assessment methods were identified and common (and unique) criteria and approaches to assessment were noted. The assessment methods studied were: the Study Design and Implementation Assessment Device (Valentine & Cooper, 2003), the Oxford rating scale (Jadad et al., 1996), the Scientific Methods Scale (Sherman et al., 1998, 2002), and criteria for study-quality assessment in the Cochrane Handbook (Clarke & Oxman, 1999). Developers of these assessment methods were contacted to identify answers to any questions regarding their approach. Experienced investigators independently applied these assessment methods to the same set of controlled and quasi-experimental studies of Multisystemic Therapy, and then compared and discussed their ratings.

Results: Findings show some overlap in criteria used to assess study quality, differing approaches (and level of specificity) with respect to assessment of specific criteria, and differences with respect to ultimate assessment of the quality of an individual study based on method used.

Implications for practice and policy: The advantages and disadvantages of specific methods for assessing study quality are delineated. The ways in which common criteria and approaches to their assessment could be used to improve and standardize the peer-review process for publishing are described. Implications for doctoral research training are elaborated.

See more of Methods of Research Synthesis: Providing Credible Evidence for Practice and Policy
See more of Symposium

See more of Celebrating a Decade of SSWR (January 13 - 16, 2005)