Assessing Risk of Bias of Psychosocial Interventions in Systematic Reviews

Background: Systematic review is generally viewed as a better way of synthesizing research evidence than traditional narrative reviews. This method has become popular in the field of social work intervention research for synthesizing treatment efficacy and effectiveness with the growing literature in randomized controlled trials (RCTs). Assessing the risk of bias of RCTs is a critical component in systematic review as recommended by Cochrane Collaboration. However, applications and reporting of the risk of bias assessment have been inconsistent in publications of systematic reviews for psychosocial interventions. It is also unclear how existing reviews incorporate the assessment findings into meta-analysis.

Objectives: 1. Describe the common practice of assessing the risk of bias of RCTs in psychosocial interventions. 2. Identify the limitations of the common practice. 3. Provide comprehensive review and recommendations on how to assess the risk of bias of RCTs in psychosocial intervention and how to incorporate the assessment findings into meta-analysis.

Methods: The author searched for systematic review and meta-analysis of RCTs in four journals that publish review articles on psychosocial interventions in the past five years. They are Clinical Psychology Review, Clinical Child and Family Psychology Review, Research on Social Work Practice, and Children and Youth Services Review. The author then extracted relevant information from each article on three main questions: 1. Did the study conduct risk of bias assessment? 2. What tool and criteria did the study use? 3. How did the study connect findings of the risk of bias assessment with meta-analysis? The author also conducted a comprehensive literature review on the limitations of existing tools.

Results: The study identified 45 systematic reviews of RCTs in the past five years from the four journals. Most of the identified reviews conducted the risk of bias assessment and employed Cochrane Collaboration’s tool of assessing the risk of bias. However, it is not always clear that how did the studies apply each criterion of the Cochrane’s tool. Examples of other used tools included Clinical Trials Assessment Measure (CTAM), modified Jadad criteria, and the modified version of the Cochrane Collaboration’s tool. The frequently used criteria of these tools do not always apply to or are not the most appropriate guidelines for assessing psychosocial interventions (e.g., blinding of participants, using intention-to-treat analysis as rule of thumb). About half of the reviews that assessed risk of bias failed to make any connections between the assessment findings and meta-analysis. Common practice of the identified reviews including limiting the quantitative synthesis only to the studies that are of ‘high quality’ in the first place, conduct sensitivity analysis by omitting studies with high risk of bias, and carry out meta-regression analysis to explore whether treatment effects vary based on the risk of bias or methodological quality.

Conclusion and implications: The existing tools for assessing the risk of bias need to be tailored for psychosocial interventions. More methodological studies are needed to explore the most appropriate approach for incorporating the findings of the risk of bias assessment into meta-analysis.