Development and Initial Evaluation of the Cyber-Counseling Objective Structured Clinical Examination (COSCE)

Purpose: As the internet has gradually become a pervasive presence in our everyday lives, cyber-counseling, sometimes called E-therapy or online-counseling, is increasingly popular (Barak et al., 2009). Given the unique provision and skills required to perform cyber-counseling, more educational institutions are offering cyber-counseling training programs. Accompanying the growth of cyber-counseling is a need for assessment measure that can demonstrate the competence level of trainees and professionals. This study described the development and pilot validation of such a tool.

Methods: The development of the Cyber-Counseling Objective Structured Clinical Examination (COSCE) involved three steps. First, we created an 18-item cyber-counseling performance rating scale based on key performance indicators of cyber-counseling (Murphy & Mitchell, 2009) and relevant practice competence measures (Bogo, et al., 2011). Second, we developed two client scenarios to be used in the piloting process. These fictional scenarios reflected typical client issues that cyber counselors might encounter. Lastly, we recruited and trained three raters to evaluate the email sessions using the cyber-counseling performance rating scale. All raters were experienced cyber counselors.

We pilot tested the validity and reliability of COSCE on six student cyber counselors and six seasoned cyber counselors. The student counselors were MSW students who have completed a cyber-counseling training workshop and participated in a yearlong cyber-counseling internship; whereas the seasoned counselors were practitioners who have a masters degree, have completed cyber-counseling certificate programs, and have practiced cyber-counseling for over 5 years. Each counselor was given the two identical fictional client emails. The counselors conducted an email session as if they were the cyber counselor for each of these two clients. The counselors were instructed to follow their typical practice as much as possible, but they were asked not to consult with other practitioners. A total of 24 cyber-counseling sessions were collected. The three raters, blind to the counselors, used the performance rating scale to assess all the sessions.

Results: Internal consistency was computed on the COSCE scores of 24 sessions assessed by all raters. The overall Cronbach’s alpha was .97. To estimate the inter-rater reliability of COSCE, we used Cronbach’s alpha to test the extent to which the three raters generated COSCE scores that corresponded. The Cronbach’s alpha was .82. Inter-client reliability was tested also using Cronbach’s alpha, and the Cronbach’s alpha was .91. To assess the construct validity, we performed a t-test to see if the COSCE correctly differentiated experienced cyber counselors from student counselors. The experienced counselors received a much higher average COSCE score (61.63 [SD = 17.15]) than the student counselors (40.94 [SD = 13.19]). The difference between the two groups was statistically significant [t(66) = 5.51, p < .0001].  

Implications: The preliminary results from this study suggest that COSCE can be a reliable and valid tool in assessing the counselor’s critical thinking, judgment, online interpersonal interaction skills, and capacity to demonstrate all these skills online. Given the growth of cyber-counseling and emerging training programs, this tool, with further replications, can be a promising addition to ensure the quality of cyber-counseling practice.