Skip to main content
SAGE
Search form
  • 00:00

    Welcome to class 8, Developing Survey Questions.Finally, we're getting around to the real work of this course,developing the survey questions that will go on to your survey.I want to start with a quote by Stanley Payne.In 1951 he said, "Why concern yourself?A plea for the importance of asking good questions."

  • 00:22

    According to Payne, too little emphasisis placed on questioner design, with the majority of attentiondevoted to sampling.He argued that improvements to questionnaire wordingand other phases of the survey research endeavorcan contribute far more to the accuracy of a surveythan tinkering with sampling.

  • 00:42

    Survey questions can range from closed dichotomous questionsto completely open-ended questions.Here's an example of a closed dichotomous question.It's very straightforward.Are you male or are you female?Here's an example of a completely open-ended question.Do you have any final comments or thoughts

  • 01:04

    that you want to share with us?And respondents are invited to write in their answers.In between the two extremes, thereare many ways to ask questions.So how do we get started?We need a few rules of thumb.I like this quote by Martha Nussbaum.She says "anything worth measuringin human quality of life is difficult to measure."

  • 01:27

    And that would probably be an understatement.So one rule of thumb is to brainstorm ideas.And this is when you want to go backto your research questions.Remember them?You want to brainstorm ideas usingall of the work you did during the conceptualizationphase of your study.Also you want to ensure that all your ideas for survey items

  • 01:48

    fits under one of your research questions.If it doesn't, you have two choices.One, remove the item.Or two, create a new question.Otherwise your survey becomes a chaotic smorgasbord.Another way to get started is to look at existing questions.You can look at questions, for example, from the censusor from other studies.

  • 02:09

    You may want to actually imply some of these questions.And doing so might be a very good idea.For example, the question on the Canadian census on occupationcomes with a whole machine of information around it.For example, it comes with a manual called the NationalOccupational Classification, which is a codingscheme to code occupations.

  • 02:30

    This would have been prepared by a team of researchers.So there's no reason to do this on your own.However, you have to ask whether questionsout there on other surveys are indicators of your constructs.They must be or else they don't work.Remember when you imply existing questions,you must be aware of copyright and related plagiarism.

  • 02:51

    Here's the definition of plagiarism by the AmericanPsychological Association.Unlike quoting text in papers, wedon't actually put quotation marks around survey items.

  • 03:12

    However you should seek permission to use survey items.And in some cases, you must seek permission.Also, you should acknowledge the source when you can.So now we're going to move on to guiding principlesfor questionnaire item design.Principle number 1: Ask questionsthat your respondents can actually answer.

  • 03:34

    Look at this question.What in the world does a combination of things mean?And what does it mean in relation to a career?Is family commitments the same as home commitments?Item A is an example of a double-barreled item,which we'll talk about shortly.

  • 03:55

    You've heard this before, survey questions are usuallylimited to behaviors, attitudes, perceptions, beliefs, facts,and demographic characteristics of the respondent.Survey research is not conducive to asking respondentsto speculate about others.Principle number 2: Ensure that each question includes

  • 04:17

    only one thought or idea.So you must ensure that both the stemof the question and the response categoriesinclude only one thought or idea,otherwise you've created a double-barreled item.An example of a double-barreled item is as follows.To what extent do you agree that your teacher was open and fair?

  • 04:40

    What if the teacher was open but not fair?What if she were fair but not open?Double-barreled items simply cannot be interpreted.Principle 3: Choose your words carefully.It's critical to use plain, simple, clear languagethat your respondents can understand.This is no time for purple prose.

  • 05:03

    Also avoid technical terms for a non-technical audience.If your audience has technical expertise, by allmeans use terms that you know that they will understand.Principle number 4: Provide instructionsto answer the question.And by this, I mean every question.So you need to tell the respondent

  • 05:23

    whether one or more response is required,the temporal period in which the response should be framed,this week, this month, this year,and you must provide definitions for terms that may nothave clear meanings, for instance,post-secondary attendance, full-time or part-time.What does that mean?

  • 05:43

    You need to provide a definition.So on to types of questions.As I've already said, there are closed-ended questionsand open-ended questions.However between the black of closed-ended questionsand the white of open-ended questions,there's a rainbow of behaviors, attitudes, preferences,

  • 06:04

    and lived experiences that you will want to capture.Here's an example of a closed-ended dichotomousquestion.Have you ever had a student loan?Yes or no.There's no other answer possible.In the next question, are you thinkingabout having children in the future?The answer could be no, it could be yes,

  • 06:25

    but it also could be maybe.Another way of expanding a closed-ended dichotomousquestion is to follow it by an open-ended question.If you could choose again, would youmake the same educational choices?No or yes.This is very efficient but it might not be very effective.

  • 06:45

    So it helps to add, please explainyour answer to this question.Respondents can use their own wordsto tell you what yes or no means.Another type of question uses unordered response categories.Here's an example.There are many response categoriesbut it really doesn't matter if medical insurance or a pension

  • 07:08

    plan is first.In contrast, this is an example of a question usingordered response categories.There are five response item, starting from not at allto more than five times a week.In this type of question, it's important to go from leastto most or most to least.

  • 07:28

    And try to have each interval between the itemsas balanced as possible.The nature of this scale is really question dependent.It will vary from question question.Rating scales build on the notion of ordered responsecategories.So we actually stack items on top of one anotherwith a common rating scale.

  • 07:49

    And of course, the scale must be the correct scale for eachof the items underneath it.Another type of question employs ranking scales.Try to do this question.

  • 08:20

    [BEEPING]Not easy, is it?I think ranking skills are highlyproblematic in terms of actually beingable to answer the question.And they're also problematic in terms of analysis.Personally, I avoid using ranking scales.

  • 08:43

    We need to take a moment to considerthe nature of the scale that we are employing.Sometimes we employ a four-point scale, for example,strongly agree, agree, disagree, and strongly disagree.This type of scale does not let the respondentsit on the fence.They must either lean towards agreeing or disagreeing.

  • 09:04

    You need to ask yourself whether this is the type of scale youwant to employ.It may also be an ethical issue.Is it ethical to force your respondentsto lean one way or another?One way around this is to have a neutral or no opinion categoryin the center of the scale.However, what does neutral mean?And what does no opinion mean?

  • 09:26

    And what happens when the majority of your respondentsanswer neutral or no opinion?Then you really don't have any data.A better way to deal with this midpoint,for example in this scale, is to say neither important or notimportant.This way it fits into the structure of the scale.

  • 09:47

    If you want to have no opinion or not applicable,put it to the right of the scale.And if you can, put it in a different colorto make it distinct from the main part of the scale.Now we need to address some be aware issues.The first is having too many agree/disagrees going on.For instance, if you ask somebody

  • 10:08

    to what extent do they agreed to the statementI am sometimes depressed, do they disagreethat I'm sometimes depressed?Do they agree that I'm sometimes depressed?It gets quite confusing.Then there are words like might, could, and should.These are speculative words that don't reallyfit into survey research, neither do concept words

  • 10:28

    like government or profit.Which government, municipal government,provincial government, federal government?You must be specific, otherwise government doesn't reallyhave a meaning.Another be aware item is that of telescoping.This means that answers to questionsare attributed to the wrong time period.

  • 10:50

    There's forward telescoping and backward telescoping.With forward telescoping, events from the pastare attributed to the time periodincluded in the question.And this leads to over reporting of events.So in the past perhaps I did a lot of physical activity.But lately, I've been more sedentary.So if I bring forward my past physical activity

  • 11:12

    to the present, them I'm over reporting my physical activity.Backward telescoping is the opposite.This is when respondents take events from the presentand move them into the past.This will result in under reporting of events.So how do we rectify telescoping?

  • 11:33

    One way is to keep your time frame as narrow as possible.Another other way is to bound your time framewithin a typical day or a typical week.That is, ask them to think about a typical dayand report accordingly.Also try memory cues, such as the first day of schoolto anchor a temporal period.

  • 11:54

    More be aware items.Vague quantifiers.What exactly does regular church attendance mean?It could mean attending church oncea year on an important religious holidayor it could mean attending every day.Be more specific.We've already talked about double-barreled items,

  • 12:16

    but they deserve a double be aware.Also double negatives are a huge problem.This example, teachers should notbe required to supervise studentsin the halls, the lunchroom, and school parking lots.Agree or disagree?What in the world could be answer to that question mean?A questionnaire is no place to ask people to conduct

  • 12:38

    an exercise in mathematics.Try answering this question.[BEEPING]

  • 12:59

    Impossible right?Sometimes we try to cram too much into a question.And this is an example.Item B asks, have you moved out of the family home?You might answer no to that question

  • 13:21

    because you actually didn't move out of the family homeor no might mean no because you weren't living in the familyhome in the first place.So we have a problem here.It's better to ask a series of questions.In a series of questions, you break downwhat you want to know into little bits.This allows you to get at what you really want to know.

  • 13:44

    In this example, I've combine both closed-ended questionsand open-ended questions to get at the idea of leaving home.[BEEPING]Also a survey is no place for loaded questions.

  • 14:06

    My favorites are questions asked by environmentalists,such as do you object to the killing of innocent whales?But would you object to the killing of guilty ones?Similarly, don't use loaded words.For example, forbid is much stronger than not allow.Choose your words carefully.

  • 14:27

    Marathon questions are a problem as well.This is when the stem goes on and on and on.It both bores and confuses the reader.Like all research, survey items canbe used to manipulate to get the answer that you want.I grew up in Northwestern Ontario.And in the summers, I used to visit my aunt

  • 14:48

    in a city called Fort William.Port William was a twin city to Port Arthur.These two cities were commonly referred to as the Lakehead.In the late 1960s, the powers thatbe wanted to amalgamate the two cities and give it a new name.However, they had a hidden agenda.To manipulate the referendum, they

  • 15:08

    constructed the following item.Would you like the new city to be named Lakehead,The Lakehead, or Thunder Bay?The name Thunder Bay earned 15,870 votes,Lakehead earned 15,302 votes.And The Lakehead earned 8,377 votes.

  • 15:34

    So the name Thunder Bay won by splittingthe vote between Lakehead and The Lakehead.This was highly manipulative because a majorityof the people wanted the new cityto be called Lakehead or The Lakehead.The very simple word "The" split the vote.It's worth taking some time thinkingabout sensitive topics.

  • 15:56

    Here's an example of one.Have you ever had a student loan?This is a question I ask on my surveys.I always add the phrase to remindthe respondent this information will alwaysbe reported anonymously.There are many other sensitive topics,such as drinking behaviors, drug usebehaviors, or sexual activity.

  • 16:18

    So you must cross these questions very carefully.And finally, there are a few more things to consider.You must order your questions in a logical, sequential way.The train of thought in the questionsmust be clear and organized.Skip questions are used to direct the respondentto the next question that they're supposed to answer.

  • 16:40

    For instance I ask, in the last five years,have you attended a post-secondary institution?If the answer is yes, they go to question two.If the answer is no, they're directedto the part of the survey that dealswith life beyond post-secondary education.You must proofread your survey for technical accuracy.

  • 17:02

    This is critical.Also they will affect your face validity.Along with proofreading as piloting.Get others to do your survey and then debrief with themto see how it went.And finally, layout, formatting, and design is critical.We'll be dealing with this in a later class.To sum up.

  • 17:23

    In this class, we've covered the following, Good survey itemdesign, the wording of questions, types of scales,and pitfalls to avoid.Now you're ready to design your survey items.

Video Info

Series Name: Designing and Doing Survey Research

Episode: 7

Publisher: University of British Columbia

Publication Year: 2015

Video Type:Tutorial

Methods: Survey research, Questionnaire design, Question formation, Sensitive topics

Keywords: accuracy; awareness; brainstorming; communication breakdown; copyright and intellectual property rights; instruction manuals; jargon; language usage; manipulation; neutrality; order effects; plagiarism; proofreading; Speculation; time factors; writing (communication) ... Show More

Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:

Keywords:

Abstract

In class 8 of her course on survey research, Professor Lesley Andres discusses the development of survey questions. She cautions against the pitfalls of vagueness, manipulation, jargon use, and difficult-to-measure items.

Looks like you do not have access to this content.

Developing Survey Questions

In class 8 of her course on survey research, Professor Lesley Andres discusses the development of survey questions. She cautions against the pitfalls of vagueness, manipulation, jargon use, and difficult-to-measure items.

Copy and paste the following HTML into your website