STACEY GIROUX:Hi, I'm Stacey Giroux.I'm in the Center for Survey Research at Indiana University.Today, I'm going to talk about a set of issuesto think about when constructing survey questionsand questionnaires.First, I'm going to give an example of howthe way that you write or ask a questionaffects, sometimes drastically, the answers you receive.I'll also briefly touch on some cognitive aspects of survey
STACEY GIROUX [continued]: methodology, and outline the processa person goes through when attemptingto answer a survey question.Then I will give you a list of thingsto consider when crafting survey questions and questionnaires.With tools like SurveyMonkey and Qualtricsavailable to just about everyone,people are fielding surveys left and right.But often, they're not very well written.It's more difficult than you think
STACEY GIROUX [continued]: to create a questionnaire that will make senseto your respondents, be easy for them to answer,and yield data that will be useful in your researchor for decision making.When I need to put together a questionnaire,I find it most useful to begin my planning and thinkingat the analysis stage of a project.What am I going to be using these data for?
STACEY GIROUX [continued]: What are my analytic goals?And what do I want to be able to say about this topicor problem?Thinking about these things in advancecan only help you create a better survey instrument.Why are question and questionnaire designso important?I'll start with an example.A study done by the University of North Carolina Survey
STACEY GIROUX [continued]: Research Unit tested two questions,which I've modified a little bit here for our purposes.Do you think the sports media treat African American athletesdifferently than white athletes?Answer yes or no.And, do the sports media treat African American athletesand white athletes the same or differently?Answer same or differently.
STACEY GIROUX [continued]: So these seem to be asking the same thing don't they?But if you actually look at the results,the researchers found that for the first question,60% of respondents answered yes, they're treated differently.But with the second question, only 40% of respondentsanswered yes, they're treated differently.
STACEY GIROUX [continued]: I don't want to talk for a minute about the questionanswering process or some cognitive aspect of surveymethodology.So there are a number of steps a person goes through when theyattempt to answer a survey question--comprehension, retrieval, judgment, and reporting.And with reporting, there can be some formatting or editingthat happens when the person delivers his or her answer.
STACEY GIROUX [continued]: Now, this is an idealized list of the stepsa person might go through in answering a survey question.And these are not necessarily an orderly progression.They can be happening simultaneously.And people can go wrong at any of these stages.So let's look at another example to illustrate this process.So here's a question that you might see on a survey,
STACEY GIROUX [continued]: or perhaps someone will call you and do a survey over the phone.Since buying this home, have you received any credit cardsin the mail that you didn't apply for?Yes or no.So first, you have to comprehend the individual wordsand put them together to understandthe question, the vocabulary, and thenthink about the reference period--here, since buying this home-- and then
STACEY GIROUX [continued]: think about credit cards versus credit card offers.Then you have to retrieve any useful informationor experiences that might help you answer that question.For example, maybe you got a credit card before Christmas,but you bought your home in early January.Then, you need to make a judgmentbased on the information you retrieved.So you got the credit card before you bought the home.
STACEY GIROUX [continued]: So that doesn't fit the proper time period.And then report your answer.And you format your response, yes or no,but you might edit your answer.Because maybe you're anxious to givea socially desirable answer-- for example,you're behind on those credit card payments.So as you can see, there's a lot going on in a respondent's headwhen they're trying to answer even just one survey question.
STACEY GIROUX [continued]: Now I'm going to move on to a few topics or areas in questionand questionnaire design.Now, this is by no means a complete list of everythingyou need to think about when writing surveys.But these are the ones that we see probably the most oftenviolated or simply not considered or ignoredwhen we review questionnaires or helppeople field their surveys.And we'll examine each of these in turn.
STACEY GIROUX [continued]: There's conceptual variability and vague quantifiers,problematic wording, response options, order effects,and recall or retrieval difficultyand estimation difficulty.And finally, we'll talk a little bit about sensitive topics.
STACEY GIROUX [continued]: Our first two items are conceptual variability andvague quantifiers.This simply means that words can have many meanings.So let's consider another example.Suessbrick and colleagues published a study in 2000,where they examined respondents' interpretations of questionsabout smoking cigarettes.They asked follow-up questions about the survey
STACEY GIROUX [continued]: after the survey itself was administered.Now, you might think that smoking cigarettes is a prettyclear subject to ask about.But as it turns out, smoking a cigarettedoesn't mean the same thing to everyone.54 percent of the respondents-- which is the majority--understood smoking cigarettes to mean even just one puff.And the rest of the respondents were evenlydivided between a meaning of cigarettes finished,
STACEY GIROUX [continued]: or cigarettes finished or partly smoked.So it's important to assume that meanings are notshared by everyone.You need to learn about those meaningsand test them before you field your final survey.And there are a few ways to do this, includingconsulting experts in the subject area,conducting cognitive interviews with a set of respondents,and pre-testing your instruments.
STACEY GIROUX [continued]: And we don't have time today to talk about all the waysthat you can do this.But testing your questions ahead of time is crucial.You'll be seeing a lot more reasonsto do this in the rest of the tutorial.Next on our list is problematic wording.And there are five aspects of problematic wordingthat I want to touch on here.
STACEY GIROUX [continued]: The first is what is known as double barreling.Think about this question that you might see on a survey--are you satisfied with our prices and customer service?The problem here is that you're ultimatelybeing asked two questions within one question.What if I'm satisfied with your prices,but not with the customer service?How is someone supposed to answer this question?
STACEY GIROUX [continued]: The second is use of negative language in a sentence-- thatis, words that mean "not" or "none" or "not any."Here's another example-- do you agree or disagreethat teens should not be fined for notobeying the local outdoor smoking ordinance?It's very difficult for the respondentto understand ultimately what you are asking in the question.
STACEY GIROUX [continued]: And including the words "agree or disagree"only confuses matters further.Keep the language simple and be clear with your question.Third is complex syntax or unclear syntax.Here's another question-- given the world situation,the government protects too many documentsby classifying them as secret or top secret.Agree or disagree?
STACEY GIROUX [continued]: The sentence could have two meanings.Either the government could be motivated by the worldsituation to protect those documents,or the government protects too many documentsthan can be justified by the current world situation.But the respondent doesn't know which meaningis the appropriate meaning.The fourth is hidden assumptions.Think about this question-- how many minutes does it usually
STACEY GIROUX [continued]: take you to commute to work?This question automatically assumesthat A, the respondent works, B, the respondenthas to commute to work, and C, that the respondent can givean answer in minutes for how longit takes him or her to commute to work.And finally, leading questions-- nowthat you've seen how you can save time,would you buy our product?
STACEY GIROUX [continued]: It's pretty clear that when someone asks this question,they're expecting a certain answer.In general, a final rule of thumbfor wording of your questions is to aim for about a sixth gradereading level.Our next topic is response options.
STACEY GIROUX [continued]: So far, we've used examples that focuson the wording of the survey questions.But you also have to think about the responsesthat you offer people.And there's actually quite a bit to consider here,and we'll look at a short list of these things.First, you need to make sure that your response options aremutually exclusive and exhaustive.This means that you make sure you offer the respondentevery possible answer they might need to answer the question,
STACEY GIROUX [continued]: and that none of those answers overlap-- that is, that they'remutually exclusive.One way to do this-- and which happensoften-- is that you offer optionsof "not applicable," or "don't know," or "other"to make sure that you've made that list exhaustive.When you're constructing response options thatare in a scale format, you need to think
STACEY GIROUX [continued]: about whether you want five options, or seven options,or maybe even more options.When you're thinking about those scales,think about whether you want to offer a midpoint-- thatis, an option in the middle of the scalethat is neither or neutral.Make sure that that scale is balanced,so that if you have a midpoint, you
STACEY GIROUX [continued]: have an even number of options on either side of the scalethat have to do with maybe agreement and disagreementor like and dislike.But it needs to be even and balanced.And make sure that you label all of those scale points.So let's look at an example that illustrates one way responseoptions can make a big difference for your data.Schwarz and Bienias conducted a study
STACEY GIROUX [continued]: testing two sets of response optionsto a question about hours of televisionwatched per week-- a high-frequency scaleand low-frequency scale.The high-frequency scale ranged from up to 10 hoursof television to more than 25 hours of television.But the low-frequency scale rangedfrom up to two and a half hours of televisionto more than 10 hours of television.
STACEY GIROUX [continued]: You can see that the proportion of respondents reportingmore than 10 hours a week of television watchingfor each target person they were asked aboutis dramatically different dependingon which scale is used, with only about 10%of the low-frequency scale respondentsreporting more than 10 hours for each of those target people,while over 40% of those who were asked the question using
STACEY GIROUX [continued]: the high-frequency scale reportedmore than 10 hours per week for each target person.Next on our list is order effects.What I mean by order effects is that the order in which youpresent questions to respondents,as well as the order in which you present response options,
STACEY GIROUX [continued]: can have consequences.I'll talk about two specific kinds of effectsfor both questions and for responses.With question order, we first have what wemight call part/whole effects.So several studies have shown that asking a specific questionabout a topic before asking a more general questionon that topic can alter answers to that general question.
STACEY GIROUX [continued]: So for example, if you want to know about marital satisfactionand general satisfaction, ask that general questionabout satisfaction before you ask about marital satisfaction.The second item under question order is related content.So fewer people say that taxes are too high if they are first
STACEY GIROUX [continued]: asked several questions about whether governmentspending should be increased for various programs.So think about all of your questionstogether, and what content you're asking about.With response option order, we have whatwe call the recency effect.And that's the tendency to endorse the last optionin a list of response options.Now, this is more likely when an interviewer
STACEY GIROUX [continued]: is reading to a respondent-- doing an interviewover the phone or in person.The primacy effect it is the opposite.That's the tendency to endorse the first optionin a list of response options.And this is more likely when the respondentis self-administering a survey-- for example, a web survey.
STACEY GIROUX [continued]: Recall and retrieval difficulty and estimation difficulty--humans are not good at remembering things accurately.Recall tasks can involve a complicated processfor respondents, and can be quite burdensome.When you ask someone a survey question thatrequires them to recall information,there are some things you can do to makethis a little bit easier for the respondent
STACEY GIROUX [continued]: and give you a better chance of getting accurate information.First of all, consider whether the respondent is evenin the position to know the answer to the question.If you ask a college student about their parents' income,will they even know the answer?Things that are more recent, that have happened morerecently to a respondent, or world eventsthat have happened more recently--
STACEY GIROUX [continued]: people are more likely to recall those.Also, events that have greater impactor greater salience for the respondent they'remore likely to recall.For example, people can very easilytalk about the birth of their children.Because that's an event that highly salient to them.In general counting, overall impressions, and hypotheticals
STACEY GIROUX [continued]: can be very difficult for respondentsto think about and answer.So perhaps use a reference periodor a point in time in your question to help jogtheir memories.Perhaps you need to get at the informationin a series of questions, rather than with only one surveyquestion.Also, you can try asking a longerquestion that uses maybe familiar words or some more
STACEY GIROUX [continued]: examples.Finally, sensitive topics-- sometimes,we want people to answer questionsabout topics that might be sensitive or threateningto them.In fact, income remains one of the most sensitive questionsfor a respondent.But in addition to simply recognizing
STACEY GIROUX [continued]: that fact, and what topics might be sensitive,there are a few ways you can try to ask questionsto make a respondent more willing to answer, and answeraccurately.Here's a less serious, but illustrative exampleof some tactics.And these are described by Bradburn,but originally done by Barton.The casual approach-- do you happen
STACEY GIROUX [continued]: to have murdered your wife?You can also try the numbered card approach,and this would work when you are doing an in-person interview.And you would hand a card to the respondent, and say,would you please read off the numberon this card that corresponds to what became of your wife?Number 1, natural death, number 2, I killed her,
STACEY GIROUX [continued]: number 3, other.The everybody does it approach-- and you might ask, as you know,many people have been killing their wives these days.Do you happen to have killed yours?And finally, the other people approach--do you know any people who have murdered their wives?How about yourself?
STACEY GIROUX [continued]: Writing good and useful surveys is not as easy as it looks.Today, we have briefly reviewed a handfulof the most important things to considerwhen writing survey questions.There is plenty to read about each of these six areas.And there are more aspects of survey designto consider that we haven't even covered here today,such as attitudinal questions, and formattingof surveys generally.
STACEY GIROUX [continued]: Here are the studies I mentioned in our tutorial today,as well as some more resources about survey designand methodology.
Publisher: SAGE Publications Ltd
Publication Year: 2017
Keywords: accuracy; ambiguity (semantics); cognitive processes; comprehension; concepts & philosophies; effectiveness; expertise; income; information retrieval; judgment (psychology); language usage; meaning (epistemology); memory; option; practices, strategies, and tools; primacy effect; reading ability; recall; Smoking; tactics; vocabulary; writing (communication) ... Show More
Segment Num.: 1
Dr. Stacey Giroux outlines issues to consider when writing a survey or questionnaire. She also describes the cognitive aspects of survey methodology and survey taking.
Looks like you do not have access to this content.
Dr. Stacey Giroux outlines issues to consider when writing a survey or questionnaire. She also describes the cognitive aspects of survey methodology and survey taking.