Research That Matters (January 17 - 20, 2008)

Purpose: In recent years, child welfare agencies have been compelled to include citizen stakeholders in public policy evaluation, and a key mechanism for this has been the development of citizen review panels (CRPs). Created through a 1996 amendment (Public Law 104-235) to CAPTA, volunteer panel members are directed to evaluate state child welfare agencies and make recommendations for improvement in services. Despite the age of this mandate, only a sparse literature regarding panel activities and their effect on services has developed. Only two studies on these panels to date have been published, both with methodological limitations. These studies' findings indicated that perceived panel effectiveness is related to increased panel/agency communication, role clarity, legitimate collaboration, and realistic goals. Factors found to impede effectiveness included mistrust, time constraints, role confusion, and weak communication. Researchers sought to assess the perceived effectiveness of the panel/agency partnership in implementing child welfare practice and system changes. Seven related research questions designed to capture multiple stakeholder perspectives formed the foundation of the study, with research activities conducted to obtain answers to these questions. These questions investigated panel members' impressions of their effectiveness; panel cohesiveness; perceived quality of panel/agency communication; agency responsiveness; panel member preparation; agency perceptions of panel roles; and agency-recommended future panel efforts. Methods: All participants recruited for the study were purposively sampled by virtue of their roles in the CRP initiative. A mixed-methods approach was utilized, including five focus groups with citizen review panels (n=34); an online survey of citizen review panel members (n=34) assessing communication flow, group cohesion, and self-governance; and content analysis of four years of CRP reports and agency responses. Findings were compared with the results of fifteen interviews with state agency personnel to obtain multiple stakeholder perspectives. Narrative data were analyzed for recurrent themes and synthesized across sources. Results: Results from this study suggest that, although the state's panels are valued by the agency, the extent to which panels fully participate with the state in implementing needed change is inhibited by several problematic factors. Panel members believed they were viewed as token partners, and perceived their ability to effect change as substantially limited. Agency interviewees agreed with this perception to some degree by indicating that developing system-wide policies were inappropriate activities for panels to undertake. Both agency administrators and the panels believed that the panels' most successful endeavors were their supportive efforts targeting local agency offices. However, this limited role does not reflect the evaluative, policymaking panel functions which are implied by the mandate. Inconsistent communication flow, the mandate's vagueness regarding panel responsibilities, agency unresponsiveness, insufficient panel training, and inadequate panel recruitment strategies were identified as reasons for perceived panel ineffectiveness and token partnership. Conclusions and Implications: This study fills a research gap by offering the results of a comprehensive, statewide citizen review panel evaluation, and also a potential prototype of how a complex public policy initiative's effects such as these may be investigated. Recommendations for improved panel/agency partnership resulting from the study may help to strengthen future CRP efforts.