// November 30th, 2011 // No Comments » // Uncategorized
Allow us to introduce you to John and Mark:
- John is intelligent, industrious, impulsive, critical, stubborn and jealous
- Mark is jealous, stubborn, critical, impulsive, industrious and intelligent
Who do you think you would like more?
It shouldn’t make a difference, since the descriptions contain exactly the same words, and yet (according to “Consumerology” by Philip Graves, Nicholas Brealey Publishing, 2010), most people unconsciously attach more weight to the words they hear first and say they prefer John to Mark.
Similarly, using neither calculator nor spreadsheet, have a guess at the answer to each of the following two calculations:
1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 = ?
8 X 7 X 6 X 5 X 4 X 3 X 2 X 1 = ?
The average guess for the first calculation, according to Consumerology, is 512, for the second 2,250 (more than four times higher). The actual answer, 40,320, is exactly the same for both. Again, people attach greater significance to the first few numbers and estimate an answer accordingly.
Both examples illustrate the dangers of research design. As Steve McKee noted*** in a BusinessWeek article:
Google once asked users how many research results they’d like to see on one screen. Since conventional wisdom says more is always better, people naturally said “more.” When Google tripled the number of results, however, it found that traffic actually declined. Not only did the results take a fraction of a second longer to load, but having more options led people to click on links that were less relevant. The respondents in Google’s research didn’t intentionally lead researchers down the wrong path; they just didn’t understand the real-world implications of their choices.
For any research to be scientifically reliable, every variable other than the one being tested must be controlled. But in most marketing research it’s impossible to control all the variables, which means a certain amount of error is in every study. Where that error lies, and how significantly it affects the outcome, is always a mystery. That’s what makes it so dangerous.