Personality tests are just surveys

Are surveys good evidence? In some cases, it may seem like there is no choice: how else are you going to find out if people say they prefer Coke or Pepsi other than asking them? Even then there are still two issues.

1. Is the topic contextual? Is filling in a survey a sufficiently unique context to tilt the answers?

2. Is there an underlying thing that can be measured?

Even something as simple as which of Coke and Pepsi you prefer can fall for these traps. Coke famously fell for it when they introduced New Coke. They responded to blind sip test results showing people preferring Pepsi. They panicked (after all, Pepsi had been growing pretty steadily) and completely re-made their formula. It was a disaster. They had to re-introduce the original Coke.

What went wrong? Well, a sip test is just a sip. Pepsi tasted better in a sip, but was too sweet in a whole can. In the context of the survey, people genuinely preferred Pepsi, but that context did not translate.

What about the second point? Is there something to measure? Is there a genuine deeper preference? I like to think I genuinely prefer Coke, but is that cultural? Does it matter? I’m sure many of you have been jumping up and down now about the claim in the first paragraph. Of course you wouldn’t ask people which they’d prefer, you’d just see what they buy more of. If that’s because of cultural training, or the ‘more complex, vanilla-based flavour’ of Coke, does it matter?

No. Because this information is used to decide what to stock. Res ipsa loquitur.

And finally, on to personality tests. These are just surveys. Complex, sophisticated surveys. But surveys all the same.

In some of the worlds I have spent considerable time, personality tests are popular. I’ve done lots of psychometric tests. At times I’ve treated them very seriously. At other times I’ve treated them like a game I can mostly ignore. I’ve seen them used as a significant part of the selection process for startup accelerators and incubators. It’s common (I’ve said it) to say that while they may or may not be accurate, they can be a placeholder for a conversation; terms we can share and discuss.

The big five model measures your extraversion (along with four other factors) and reports your percentile. Using the description of extraversion implied by the questions in the survey, and validating with psychologists that this is what they meant, we can observe people and rate their actual behaviour against their measured extraversion.

This is the sort of thing that the originator of much psychometric testing, Francis Dalton, imagined.

People have done this. And found that personality is highly contextual. People’s extraversion (as measured from behaviour) changes wildly, depending on the situation they are in.

Is completing a personality test getting a real measure of your personality; something that will emerge in other situations; a simulated context that is replicated? Or, is completing a personality test an almost entirely unique context, only measuring the personality when taking a personality test? Is that any use?

I’m going to stop using personality tests as a placeholder for a conversation. Instead I’m going to be talking about the specific, observable behaviours. Those observations are not exactly facts, but they’re a lot closer than loaded language inferred from a survey.

But, back to that second point, right at the top. Are these tests meaningfully measuring anything? If the results determine access to an accelerator, maybe they’re not harmless toys. Is there a core personality that can be consistently described by language? Or is it all learned responses to contexts?