Growing Pains of Virtual Data Integrity in Measuring Discrimination: Lessons Learned from an Intersectional Mixed Methods Study with 2SLGBTQ+ People

Background and Purpose: There has been an increase in intersectional 2SLGBTQ+ research addressing the experiences of different subgroups (e.g., racialized trans people), including studies using online surveys. Online surveys can help to recruit large diverse samples and engage hard-to-reach groups (e.g., those not out). Despite their benefits, ensuring data integrity in online surveys is challenging, especially given the risk of fraudulent respondents. They can also be limited in terms of integrating an intersectional lens, including assessing discrimination in an intersectional way and operationalizing McCall’s (2005) intercategorical complexity. Drawing on the implementation of the Wellbeing & Inclusion in Rainbow Leisure Study (WIRLS), this presentation reports on these issues and offers some solutions for researchers to consider in their studies.

Methods: WIRLS used a sequential explanatory mixed-methods design to examine 2SLGBTQ+ people’s experiences and outcomes in 2SLGBTQ+ leisure spaces (e.g., gay sport leagues, trans chatrooms). Participants from Canada and the US were recruited via social media and 2SLGBTQ+ organizations using purposive and quota sampling (e.g., 100 trans) for an online survey ($10 incentive; n=548; 49.1% Canadian; 55.5% trans; 34.1% racialized; 49.3% disabled) and semi-structured interviews ($50 incentive; n=22; 36.4% Canadian; 63.6% trans; 68.2% racialized; 50.0% disabled). Assessing intersectional discrimination was a priority; Scheim & Bauer’s (2019) Intersectional Discrimination Index focusing primarily on direct/personal discriminatory experiences was adapted for the 2SLGBTQ+ leisure context with additional items developed in collaboration with WIRLS’ advisory committee. To understand intersectional discrimination among various subgroups, an intercategorical analysis was conducted using two-way and three-way interaction terms with gender, race, and disability.

The survey included data integrity strategies (e.g., multiple choice and open-text quality control questions, Captcha). However, additional tactics were needed given the suspicious responses upon initial launch; a separate screening survey and metadata functions in Qualtrics were implemented, and every screening survey response and the metadata were reviewed to judge the validity of submissions.

Results: All 167 responses submitted upon the survey’s initial launch were deemed to be fraudulent or bots (e.g., IP address outside North America, nonsense open-text responses). With the additional data integrity strategies, approximately 6% of responses were classified as fraudulent. Scores on the intersectional discrimination scale were relatively low (M = 0.36, SD = 0.50; theoretical range of 0-3), but interviewees, including those reporting little discrimination on the survey shared many experiences, including indirect/ambient discrimination targeting other members. Despite the relatively large sample, group size issues arose when examining three-way intersections of identities (gender X racialized X disability) to reflect the intercategorical complexity. Qualitative analysis identified gender identity/expression, race, and disability and associated oppressive systems as influential in shaping participants’ experiences in 2SLGBTQ+ leisure spaces.

Conclusions and Implications: The additional strategies employed in the survey minimized fraudulent respondents and bots, thus promoted data integrity. Though the Intersectional Discrimination Index is useful, it is important to address indirect/ambient discrimination to holistically understand discrimination. Given the inability to interact various positionalities in survey research, mixed-methods research can address this and provide insights into the complexity of intersectional discrimination.