Who's the Expert? Sexual Minority Youth Define Critical Ingredients for Implementation Fidelity In Strengths-Based Case Management

Purpose: Implementation fidelity is a critical but overlooked part of evaluation research. Obtaining a youth participant perspective has eluded sufficient capture in fidelity studies (Checkoway & Richards-Schuster, 2003) yet may be relevant to social work research in community-based programs. The ability of vulnerable multiethnic sexual minority youth (MSMY) to identify components of Strengths-Based Case Management (SBCM), and understand the intent of the model can enhance implementation fidelity (Resnick, et al., 2005), improve the design and delivery of the program, as well as provide evidence for model effectiveness. While practice experts have defined the critical elements of SBCM (Marty, Rapp, & Carlson, 2001), this study suggests that MSMY recipients also possess expertise through the description the important elements of SBCM, and rating the content validity of SBCM, producing fidelity criteria.

Methods: In a mixed-method approach, three primary steps were addressed. (1) Based on the SBCM literature, a 49-item checklist was generated, (2) focus groups of MSMY participants were conducted and analyzed and (3) a content validity index (CVI) was developed through MSMY expert ratings. The checklist was developed using the 81 items from the original study of SBCM critical elements (Marty, et al., 2001) and items drawn from the literature on client perspectives of SBCM (Brun & Rapp, 2001). Focus groups were conducted with MSMY expert alumni (n=17) of a SBCM program to gather the important receipt elements of the model in the participant's language. Three independent coders (inter-rater reliability=0.92) analyzed emerging themes using behaviorally anchored criteria (Mowbray, Holter, Teague, & Bybee, 2003). MSMY experts (n=23) established content validity (Lynn, 1986) by rating each item on 3-point Likert scale of importance. Average importance scores were computed for comparison to Marty et al. (2001), and a CVI of the proportion of experts rating an item as very important was calculated.

Results: Themes arising from the focus groups included: 1) importance of the relationship to the case manager, 2) importance of talking about strengths, 3) being a partner in planning, and 4) having an advocate. From these themes, 3 additional items were added to the checklist. For each item an average importance rating across all experts and a proportion of experts rating the item as very important was calculated. For example the item, “It's important that my worker cares about me”, scored an average of 2.8/3 (>2.5 as important to the model), and a CVI score of 93%. Items with a score >2.5/3 (Marty, et al., 2001) and a CVI score of greater than 80% (Lynn, 1986) were retained as fidelity indicators (n=20).

Conclusions and Implications: This study demonstrates that vulnerable MSMY youth can be expert consultants in an evaluation design to measure implementation fidelity. The final tool elicits participant perspective on the service model, and identifies the presence of critical elements. The developed tool provides legitimacy to the model (Walker, 2007), informs the design and delivery of the service, and provides data on implementation fidelity to enhance validity and reliability of evaluation findings.