Recognizing and Mitigating Bot Infiltration in Online Survey Research

Background and Purpose

This poster reports on the methodological challenges of conducting survey research in the age of artificial intelligence. The internet makes it easy for researchers to recruit participants and distribute surveys. However, it is susceptible to attracting the attention of scammers who use computer-generated algorithms, in this case survey bots, to infiltrate online surveys for the purpose of collecting incentives offered in exchange for completed surveys. The purpose of this poster presentation is to inform researchers of the risks and consequences of online recruitment and data collection and to offer strategies to minimize the potential for, and respond to interference by, survey bots.

Methods

At the heart of our research is a collaboration with a local youth-homelessness agency and a statewide homelessness consortium to conduct a needs assessment. We created an electronic survey aimed at young people who were unstably housed or living on the streets. Eligible youth were invited by agency staff to complete an anonymous online survey for which the youth would receive a $15 prepaid Visa gift card via email. To claim the incentive, respondents were required to reach the end of the survey, where they would click a link to a second, separate survey that collected their name and email address.

Results

After receiving very few responses, the agency promoted the survey more widely. By the next day, the main survey had received 134 responses and the incentive survey 2,464. Since we were checking the responses only weekly, the exponential increase initially went unnoticed. Before long, the main survey had received nearly 8,000 responses and the incentive survey had received more than 9,000. Almost all (87.5%) of the main surveys were at least 98% complete. The incentive survey recorded 8,856 unique email addresses, all Gmail (which are harder to trace than other email addresses), and all of which had been automatically created by the survey bots. In addition to the inordinate number of completed surveys, we noticed other tell-tale signs of inauthentic responses, such as implausible response times and identical qualitative responses. Strategies to reduce fraudulent responses include

1. distributing surveys through known networks and not social media
2. requiring respondents to request the survey link
3. using a unique survey URL for each respondent
4. using task-based tools (e.g., anagrams) and question-based tools (e.g., feeling screeners)
5. rejecting survey cases in which the respondent failed at least three detection tools
6. considering commercial products that detect IP addresses and virtual private networks/servers

Conclusions and Implications

Artificial intelligence continues to make a significant, positive impact on social science research, but its capabilities are also causing considerable disruption by making survey bots more sophisticated and human-like. Bots infiltrate online surveys, resulting in thousands of fraudulent (yet very realistic) responses that infect datasets, rendering them useless. Social work and other social science researchers must be aware of the potential threat to their data and employ these and other strategies to prevent or mitigate the damage that survey bots can and do cause.