“Believe none of what you hear and only half of what you see.”
It’s a great rule to live by, and, it points out one of the main dangers inherent in market research, or research of any kind, really. Good research requires good filters and a high level of skepticism. You see, what research subjects say, and even what they do, don’t automatically tell us what they really think or will really do once released from the confines of the artificially created research environment.
Don’t get me wrong. I believe there is a place for research –it can be extremely valuable. I’m just warning you to not take the results of any research, formal or informal, at face value.
First, as any good journalist will tell you, you must consider the source.
If a four-year-old child draws a picture and then asks how you like it, what will you say? Unless you are a heartless beast, you will probably tell him or her it is fantastic. So, when you describe to your spouse, partner, employee or mother the idea you came up with for a new ad campaign, what makes you think they will say anything other than, “That’s fantastic!”? And, unless they are a marketing expert or squarely in your target market, why should you care what they think anyway? In research, we call this a convenience sample – a sample of the population that was easy to access. Convenience samples are almost totally useless.
So, what’s a better way to test an idea or concept – a formally structured research methodology that leverages the power of truly random samples to explore the depths of consumer motivation? OK, sounds good, in a business-school kind of way. But how do we get a random sample? Oh, wait. I know. We’ll hire someone to call random people out of the phone book and ask them a series of questions related to the topic we are trying to probe. That’s random, right?
Wrong.
There are a bunch of problems with calling this kind of sample “random.” First, it’s only a sample of people who have phones – and fewer people (especially young people) have them at home these days. Then, our “random” people must have their phone number listed in the book, be home when we call and be willing to answer. Hmmm, this is starting to sound like a sample of elderly, talkative, lonely shut-ins.
The same limitations apply to online surveys, face-to-face questioning in public places and even interviews or focus groups composed of subjects recruited by professionals. None of these provides perfectly random samples and all come with inherent limitations and biases.
So, what’s the answer?
The answer is: Believe none of what you hear – not at face value, anyway – because there are no truly random samples and because all of us have inherent biases. All subjects come with challenges. Therefore, all data collected from them must be evaluated skeptically, professionally and with an understanding of the source and the situation.
So, wouldn’t it be better to emulate Gorillas in the Mist naturalist Dian Fossey, who spent years hiding behind trees in Africa watching mountain gorillas in their natural habitat and writing down everything they did? Couldn’t we better predict consumer behavior by watching them?
Maybe.
But what if our presence – our watching them – somehow changes their behavior. Have you ever walked into a room full of people and felt the mood change? Maybe they were talking about you. Or maybe, because you are their boss, they felt like they should behave in a certain way when you were present. It’s no different with research subjects. Do you really think the members of your last focus group didn’t know you were watching them from behind the huge mirrored wall? And do you really believe they weren’t a little intimidated knowing that?
So, we must also believe only half of what we see.
OK. This is starting to make research sound like a waste of time and money, which it definitely is not. My point is, like so many things in business, conducting and interpreting research in meaningful ways requires training, experience and insight. It’s tempting to try to find shortcuts, but that is usually false economy. Making decisions based on bad or incomplete research usually results in missed opportunities, half-baked strategies and wasted time and energy. So, the next time you think you have a great idea, don’t ask your mom what she thinks. Ask a pro to help you find out for sure.
