I wrote this for the Birmingham MA TESOL mailing list: you might find it interesting...
A long e-mail, but I hope I can make an interesting point about interview questions using an example from politics without being political.
In the US, there has been some interesting reporting on people not believing that President Obama was born in the US. Billionaire, and potential Presidential candidate, Donald Trump made a lot of news in April for publically stating that he too doubted that the President was born in the US. Many different polls have been done about what people think about the President's birthplace, but I would like to point you to this very interesting part of a USA Today article that I think brings up an important issue for anyone interested in doing questionnaire or interview research:
"Still, in the USA TODAY poll, only 38% of Americans say Obama definitely was born in the USA, and 18% say he probably was. Fifteen percent say he probably was born in another country, and 9% say he definitely was born elsewhere.Republicans are inclined to say the president was born abroad by 43%-35%.For what it’s worth, not everyone is convinced Trump was born in the USA either: 43% say he definitely was born here, and 20% say he probably was; 7% say he definitely or probably was born in another country. Nearly three in 10 say they don’t know enough to say." http://tinyurl.com/3db46w5
The take-away point being reported is that only 38% of people believe Obama was born in the States, but the real story is that people don't believe that Obama AND Trump were born in the States.
Why is this? Do they both seem incredibly foreign to most Americans? Are Americans just stupid?
Well, as an American, I won't comment on the last bit, but I do think things are bit more complicated than they appear and this poll exposes some very real issues for researchers like ourselves.
It seems to me that this poll shows a few things:
First, whenever you ask a question, you prime a response. For example, if I ask you, 'What did you have for lunch today and was it good?' I will get a response about what you ate for lunch and how it tasted. This might seem like an obvious statement, but think about it in relation to another question: 'What did you eat today?' or more broadly, 'What did you do today?' The chance of hearing about your lunch will diminish the further you move away from a primed topic. This can be a good, bad, or neutral thing, but if you look at the poll above, I think it might be evidence that the asking of the question is important. What if the pollster had asked, 'What are 5 things you don't know about Obama/Trump?' or 'Tell me what you know about Obama/Trump.' Of course, the pollster won't ask this because the results won't lead to a simple, take-away fact that everyone will clearly recognise and remember, but the answers might tell you more about what people actually think.
Second, highlighting lack of knowledge in one specific area might just be highlighting a general lack of knowledge. Although you might assume that someone was born in the country that they're from, it's not something you generally know about people you aren't close to. Come to think of it, I don't know where most of my friends were born... I assume they were born in the country they are citizens of, but they could have been born somewhere else. If you ask me about any one of them, I would probably say, I assume they were born in the country they're from, but honestly, I don't know that they were. I actually only know what town my wife was born in from filling out landing cards for immigration, when I had to ask.
Third, the question might be placing doubt in people's minds just by asking it. Think of it this way (back to our lunch example). What if I ask you, 'How do you feel this afternoon, after having had lunch in the school refectory?' or 'There was a report of food poisoning in the school refectory today. How does your stomach feel?' You can see how you might pause before you answer… 'Come to think of it...' If your question implies a problem, I will probably think there is a problem.
One of the best ways to help you deal with issues like these (or at least identify potential problems with priming) in questionnaire research is to make sure you pilot your questionnaire first and interview your pilot respondents before revising the questionnaire and giving it to a larger group of respondents. This can help you weed out some problems. You might also find that broad interview schedules or longitudinal observation of practices might be a more useful tool than questionnaires, as they help you get respondents talking broadly about topics, rather than forcing them into a fixed set of questions. Of course, all of this depends heavily on what you want to know, but mixed methods will almost always give you more perspective on your data, even if you don't ultimately end up using all the information you've gathered.
Hope this is helpful to some. Thanks for your time, as always!