Free lesson: Who is really answering your survey?
For decades, survey research rested on one assumption: if a response is coherent, a person wrote it. That assumption no longer holds. This lesson shows who else is answering your survey, and what they do to your results.
Five kinds of bad responses
| Type | Who or what | Typical signs |
|---|---|---|
| Bots | Scripts that fill in forms automatically, usually to collect incentives | Very fast completion, random or identical answers, nonsense text, filling in hidden fields |
| Fraudulent humans | Real people, sometimes organised in groups, submitting many entries or faking eligibility | Repeated payment details or devices, near-identical open answers, locations outside the study area, agreeing with everything |
| Careless respondents | Genuine participants who stop reading | Speeding, the same answer down a whole scale, failed attention checks, one-word answers |
| AI-assisted respondents | Genuine participants who paste answers from a chatbot | Long, polished, generic open answers; pasted text |
| AI agents | Software that completes the whole survey with a consistent fake persona | Very few: they are built to look human |
What the research now shows
- Crowd workers use AI. When Veselovsky, Horta Ribeiro and West (2023) re-ran a summarising task on Amazon Mechanical Turk, they estimated that 33–46% of workers had used a large language model to do it.
- Survey participants use AI for open answers. In a study published in Sociological Methods & Research, 34% of participants recruited from a popular online platform said they had used AI to help answer open-ended questions. AI-written answers were also more uniform and more positive than human ones, which can hide real differences between people (Zhang, Xu & Alvero, 2025).
- AI agents pass attention checks. Westwood (2025, PNAS) built an autonomous "synthetic respondent" that kept a consistent persona and passed 99.8% of standard attention checks across 6,000 trials. The paper argues that most current detection methods are no longer enough for unsupervised online surveys.
Not everyone agrees about the scale of the problem: critics point out that Westwood showed what is possible, not how common agent attacks are in real surveys. Both sides agree on the practical conclusion, though. Checks that worked five years ago are not enough on their own, and researchers now have to plan for data quality from the first day.
What bad responses do to results
In this course you will clean a practice survey of 300 responses about online learning. Some of them are bots, fraudulent, careless or AI-assisted, mixed in with genuine answers. Analysed without cleaning, it gives these results:
| All 300 responses | After cleaning | |
|---|---|---|
| Reliability of the 6-item satisfaction scale (Cronbach's α) | .75 | .86 |
| Correlation between satisfaction and weekly study hours | r = −.08 | r = .10 |
The correlation changes direction. A researcher using the raw data would report that students who study more are slightly less satisfied. Neither result is significant in this small example, but in a real study the same distortion can create or hide a finding. Bad responses are not random noise that averages out.
Who is most at risk
- Surveys with a public link posted on social media, forums or flyers.
- Surveys that pay a guaranteed incentive for every completion.
- Surveys on low-barrier online platforms with no identity checks.
- Studies whose results matter: grant evaluations, theses, policy surveys.
Lower-risk designs (a class you teach, a staff list, a verified panel) still have careless respondents, and increasingly AI-assisted ones.
Before the next lesson
Think of the last online survey you ran or plan to run. How were people recruited? Was there an incentive? Was the link public? Note your answers: Lesson 2 shows how to close each gap.
References: Veselovsky, V., Horta Ribeiro, M., & West, R. (2023). Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv:2306.07899. · Zhang, S., Xu, J., & Alvero, A. J. (2025). Generative AI meets open-ended survey responses: Research participant use of AI and homogenization. Sociological Methods & Research, 54(3), 1197–1242. · Westwood, S. J. (2025). The potential existential threat of large language models to online survey research. Proceedings of the National Academy of Sciences, 122(47), e2518075122.
This is Lesson 1 of Clean Data in the Age of Bots
Five more lessons, a 300-response practice survey with an answer key, and the Data Integrity Toolkit are in the full course.