Lesson 1 · about 20 minutes · free preview

For decades, survey research rested on one assumption: if a response is coherent, a person wrote it. That assumption no longer holds. This lesson shows who else is answering your survey, and what they do to your results.

Five kinds of bad responses

TypeWho or whatTypical signs
BotsScripts that fill in forms automatically, usually to collect incentivesVery fast completion, random or identical answers, nonsense text, filling in hidden fields
Fraudulent humansReal people, sometimes organised in groups, submitting many entries or faking eligibilityRepeated payment details or devices, near-identical open answers, locations outside the study area, agreeing with everything
Careless respondentsGenuine participants who stop readingSpeeding, the same answer down a whole scale, failed attention checks, one-word answers
AI-assisted respondentsGenuine participants who paste answers from a chatbotLong, polished, generic open answers; pasted text
AI agentsSoftware that completes the whole survey with a consistent fake personaVery few: they are built to look human

What the research now shows

  • Crowd workers use AI. When Veselovsky, Horta Ribeiro and West (2023) re-ran a summarising task on Amazon Mechanical Turk, they estimated that 33–46% of workers had used a large language model to do it.
  • Survey participants use AI for open answers. In a study published in Sociological Methods & Research, 34% of participants recruited from a popular online platform said they had used AI to help answer open-ended questions. AI-written answers were also more uniform and more positive than human ones, which can hide real differences between people (Zhang, Xu & Alvero, 2025).
  • AI agents pass attention checks. Westwood (2025, PNAS) built an autonomous "synthetic respondent" that kept a consistent persona and passed 99.8% of standard attention checks across 6,000 trials. The paper argues that most current detection methods are no longer enough for unsupervised online surveys.

Not everyone agrees about the scale of the problem: critics point out that Westwood showed what is possible, not how common agent attacks are in real surveys. Both sides agree on the practical conclusion, though. Checks that worked five years ago are not enough on their own, and researchers now have to plan for data quality from the first day.

What bad responses do to results

In this course you will clean a practice survey of 300 responses about online learning. Some of them are bots, fraudulent, careless or AI-assisted, mixed in with genuine answers. Analysed without cleaning, it gives these results:

All 300 responsesAfter cleaning
Reliability of the 6-item satisfaction scale (Cronbach's α).75.86
Correlation between satisfaction and weekly study hoursr = −.08r = .10

The correlation changes direction. A researcher using the raw data would report that students who study more are slightly less satisfied. Neither result is significant in this small example, but in a real study the same distortion can create or hide a finding. Bad responses are not random noise that averages out.

Who is most at risk

  • Surveys with a public link posted on social media, forums or flyers.
  • Surveys that pay a guaranteed incentive for every completion.
  • Surveys on low-barrier online platforms with no identity checks.
  • Studies whose results matter: grant evaluations, theses, policy surveys.

Lower-risk designs (a class you teach, a staff list, a verified panel) still have careless respondents, and increasingly AI-assisted ones.

Before the next lesson

Think of the last online survey you ran or plan to run. How were people recruited? Was there an incentive? Was the link public? Note your answers: Lesson 2 shows how to close each gap.

References: Veselovsky, V., Horta Ribeiro, M., & West, R. (2023). Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv:2306.07899. · Zhang, S., Xu, J., & Alvero, A. J. (2025). Generative AI meets open-ended survey responses: Research participant use of AI and homogenization. Sociological Methods & Research, 54(3), 1197–1242. · Westwood, S. J. (2025). The potential existential threat of large language models to online survey research. Proceedings of the National Academy of Sciences, 122(47), e2518075122.

This is Lesson 1 of Clean Data in the Age of Bots

Five more lessons, a 300-response practice survey with an answer key, and the Data Integrity Toolkit are in the full course.

See the course (US$59)