Are surveys good evidence?

We’ve all taken surveys. There’s one right here on this page. But, are we all aware of how much modern knowledge is derived (through statistics, usually) from answers to surveys? How accurate and reliable is that knowledge?

The pseudonymous blog, Carcinisation, has published a long essay by a literal banana on surveys. It’s worth reading in full If you’re interested in surveys and what can be derived from them. I’m still digesting. But this immediately seems relevant to Pandita’s ongoing mission to provide high-quality evidence of how we help learning.

There is a lot of knowledge derived from surveys. A lot. It’s particularly common in the social sciences, but the social sciences do overlap with other areas. Just today I saw a study announced looking into what makes engineering teams productive. And it uses surveys. Plus a mere 15+ hours of interviews.

Surveys have obvious quality problems. The banana’s essay lists four core issues. Think about the last survey you filled in when you’re reading through this list.

1. Comprehension. Does the completer understand what the question means? Do they even understand the words being used? Do they have the same understanding of those words as the author?

2. Attention. How much attention is the completer paying? Is that consistent throughout the survey? Is it consistent across completers?

3. Sincerity. Is it being completed seriously? Are they having a bit of fun? When delivering questions in-person you can give them a look when they start messing with you. How does that happen for anonymous surveys completed online?

4. Motivation. Has the completer discerned some deeper meaning behind the questions? Or behind the questioner? Are they answering the questions, or are they addressing that motivation? Have they understood the correct motivation?

If that’s not enough, here’s a tricky question: is there actually a thing to be measured? The strong interaction held atoms together long before humans were aware it existed. There was always something there to measure; we just had to become sophisticated enough to measure it. A survey is supposed to measure something. Does that something even exist?

In the world of education, this shouldn’t be an issue. Something is being taught, that something is to be understood. How well understood is it?

But, personally, after reading this (twice!), thinking about my survey taking habits, and where I’ve seen surveys used, I’ll be highly sceptical of discoveries that could only have been determined with surveys. Developer productivity and behavioural studies, I’m looking at you.

Except… there are some things that can only be discovered through surveys. Oh no!