Youth of Color Perspectives on the Development of Community-Based Survey: Using Cognitive Interviews to Evaluate Items for Measuring Racial, Ethnic, and Gender Identity, Social-Emotional Development, and Program Social Environments

Background. Developing a youth survey that is applicable to populations of youth from multiple racial and ethnic backgrounds is challenging given the limitations of existing measures of positive youth development. Specifically, measures of racial and ethnic identity have been largely tested with a single racial or ethnic group, and assess either racial or ethnic identity development, often conflating the two. Additionally, measures of social-emotional development lack consideration for cultural differences, ignoring the role of structural factors. Due to these limitations, we conducted a series of cognitive interviews with youth of color to help inform the development of a youth survey being designed to evaluate a county-wide initiative that aims to improve the functioning and well-being of youth ages 5-24. With Intersectionality as a guiding perspective, we sought to better understand these youth perspectives concerning (1) the cultural responsivity and developmental appropriateness of the identified survey questions, (2) the range of information the survey questions elicit, and (3) how youth make judgements with respect to their answers.

Methods. We conducted three rounds of cognitive interviews with a convenience sample of 43 youth of color aged 11 to 22. A sample of five organizations serving racially and ethnically diverse youth between the ages of 11-24 were first identified, then organizational staff identified potential youth participants. Participants identified with a variety of ethnic (i.e., Cham, Chicano, Cambodian, African American and African) and racial groups (i.e., Black, Multiracial, Latino/a, and Pacific Islander), were mostly male (58%), and primarily identified as straight/heterosexual. Interviews lasted 30-60 minutes, conducted at each organizational site, and participants received a $25 gift card for compensation. Thirty-nine of the 43 interviews were were audio recorded with participants permission and transcribed, and detailed notes were developed for interviews not recorded. The cognitive interview data were analyzed in dedoose using inductive analytic techniques that included open and axial coding and the constant comparative method.  

Results.Three main themes related to the issues tested by cognitive interviews arose: (1) vague and confusing questions, (2) varying interpretation of terms and definitions, and (3) mismatch between survey questions and response options. Specifically, some young people struggled with the definitions provided for racial and ethnic identity or found it difficult to understand the differences, while others found these difference supremely important given their racial and ethnic identity. For example, youth who identified as ethnically Somali found the difference with being racially Black to be extremely important. This tension was reflected in the need to balance the literacy levels and nuance of survey questions, as development and literacy levels varied among participants. Additionally, what was incomprehensible to some, offered important nuance to others especially with respect to the definitions and questions for racial and ethnic identity.

Conclusions. Young people’s input is critical to developing a survey that is meaningful and well understood by young people from highly diverse racial and ethnic backgrounds. Lessons learned about the use of cognitive interviewing techniques in a community based sample of diverse young people will be shared.