Skip to main content
Search form
  • 00:01


  • 00:11

    LILIAN YAHNG: Hi, my name is Lilian Yahngand I am Senior Methodologist at the Center for SurveyResearch at Indiana University.Now what is a methodologist, you might be wondering.Well, I spend most of my time reviewing and constructingresearch proposals.So in other words, designing studies and thinking outthe best methods to ensure rigorous and scientifically

  • 00:32

    LILIAN YAHNG [continued]: valid survey data.This presentation will give you an overviewof important methodological considerations for youto think about in designing and evaluating, or participatingin, survey projects.I will be covering the following points-- whatis survey research, how to design a good survey researchproject, and participation in surveys.

  • 00:54

    LILIAN YAHNG [continued]: But first, why should you care about any of this?You may not realize it, but you live in a world constructedin part by data from surveys.If you brushed your teeth this morningor took public transportation or heard on the news last nightthe national unemployment rate, chancesare you are a consumer of survey data.

  • 01:14

    LILIAN YAHNG [continued]: And you've probably encountered a survey or two yourself.Maybe your school sent you one via email.Or sometimes you see websites, like YouTube,asking whether you've heard of this or that brand.Or maybe you've seen someone at your local farmers marketwith an iPad going around asking people questions.Or if you're very, very lucky, youget a call asking you to participate

  • 01:35

    LILIAN YAHNG [continued]: in academic research project from a researchcenter like mine.A survey is a survey is a survey, right?Well, not quite.Just because there's a survey of some sortdoesn't mean it's survey research.By that I mean to distinguish it as an activity

  • 01:57

    LILIAN YAHNG [continued]: of systematic inquiry.Survey research isn't just getting at characteristicsof just some individuals here and there--or even only of individual respondents who took a survey.Rather, the idea is to take those individual surveyresponses and make inferences to a larger group or population.Ultimately, the aim of a lot of survey research

  • 02:19

    LILIAN YAHNG [continued]: is to get statistical estimates on characteristicsof that larger population.And that is how you get somethinglike a national unemployment rate--based on interviews with respondents in a samplesubset of the population.So while people are often familiar with surveys,and perhaps sometimes more familiar with avoiding surveys,

  • 02:41

    LILIAN YAHNG [continued]: some may not realize that there is actuallya science behind them.And that's what we'll be talking about today.The science of surveys.Let's say you need data on a topic.What should you do?Before you start, why don't you take a step back and ask

  • 03:02

    LILIAN YAHNG [continued]: yourself some questions.One, is a survey the most appropriate methodof collecting the data that I need?In other words, should I even do a survey?What kind of data do I need?Do you need, perhaps, factual or objective information?In that case, records may be your best bet,because those records may be more accurate than people's

  • 03:23

    LILIAN YAHNG [continued]: memories.Maybe what you need are behavioral data.In which case, if you want to tell a causal story,maybe you should run a clinical or an experimental study.Or maybe what you need are observational data.For instance, if you want to run a study about children'sinteractions at school.Or maybe what you need are data about beliefs, attitudes,

  • 03:43

    LILIAN YAHNG [continued]: or what people know.And here what you want to ask is, are those datathat you want to end up with goingto be mainly quantitative data or qualitative data.All of these kinds of data can be collected by a survey.But don't automatically assume a surveyis the best way for getting what you need.

  • 04:04

    LILIAN YAHNG [continued]: Also you need to ask yourself, is therean existing survey questionnaire or data set out there already?You should really explore the wealthof data and questionnaires already available,either publicly or through, say, a school subscriptionto a repository.The Inter-university Consortium for Political and SocialResearch is a great place to start,

  • 04:25

    LILIAN YAHNG [continued]: as are the US Census and the Center for Disease Control.But let's say that you are set on doing a survey project.Congratulations, now prepare for some hard work.Let's look at what you're in for.This is a chart of the survey process.You read this chart from top to bottom,and that's how we're going to go through it today.

  • 04:47

    LILIAN YAHNG [continued]: First off, you want to define your research objectives.What is your research question?You have to think hard about this.Then you have two major decisionsthat you need to make.One, you need to choose the mode of data collection,and you also need to choose a sampling frame.A sampling frame, you can think of as a kindof a list of contact information,

  • 05:08

    LILIAN YAHNG [continued]: like an email distribution list.Now you'll also need to choose, like I said,the mode of data collection.The mode of data collection is the method by whichyou deliver the survey.That is, an online survey, a phone survey,an in-person survey, a mailed paper surveyor an in-person paper survey, or a combination of all of those.Now there's some interaction between the mode

  • 05:29

    LILIAN YAHNG [continued]: of data collection and your chosen sampling frame.So let's say that you've made these two major decisions.What you need to do then is to, on the sampling side,design and select a sample.That's what we call sampling.And on the other side, on the left-hand side, whatyou need to do is after you have a chosen mode of datacollection, you need to construct and pre-test

  • 05:52

    LILIAN YAHNG [continued]: your questionnaire.After those two things, you'll go aheadand do data collection.That is, you'll recruit and measure your sample.And you'll wait for all the responses to come in.And once you get all of your survey responses,you'll go ahead and code and edit the data.This is what we call data processing.After that, you might want to make

  • 06:13

    LILIAN YAHNG [continued]: some post-survey adjustments.This, too, is data processing.And finally, the fun stuff.You'll perform some analysis.As you can see, good surveys take time and care.It isn't just anything goes, making it upas you go, willy nilly.And it isn't the work of a weekend.If you are serious about conducting a survey,

  • 06:33

    LILIAN YAHNG [continued]: a well-designed survey, here are two driving questionsto help start you off.One, what are my analytic objectives?The survey process chart starts with a question-- your researchquestion.And ends with, hopefully, an answer to that.But when you are thinking about your research question,also think about what analyses you'llneed to run to get your answer.

  • 06:54

    LILIAN YAHNG [continued]: That can impact multiple points in the design of your survey--from the survey questions to the method of survey modeand sampling.Second, what resources are available to me for samplingand data collection?What you may want to do is a general population study,gauging opinions on, say, global warming.But this can be a very expensive undertaking.

  • 07:15

    LILIAN YAHNG [continued]: All surveys are constrained by cost.You need to assess, very practically,what is possible for you.Now speaking about quality within cost constraints,there is a central guiding document in the survey researchfield that does just that.

  • 07:35

    LILIAN YAHNG [continued]: Let me introduce you to what is called, total survey error.You might think of it as a kind of cheat sheet of all the waysa survey can go wrong-- how the data you end up withcan be problematically deviating from the truth.And that is what we mean by error in this context.It isn't an error in the sense of a mistake or accident,but rather error means a deviation from the true value.

  • 07:59

    LILIAN YAHNG [continued]: So more precisely, total survey errordescribes potential sources of systematic errorin sample surveys.And in doing so, it serves as a framework for conceptualizingsurvey quality.Also it is worth noting here that, strictly speaking,total survey error pertains to sample surveys.That is, surveys that get their datafrom a subset of a population.

  • 08:20

    LILIAN YAHNG [continued]: But it's basic principles can be used as a helpful guidelinefor almost any survey project as a way of maximizing accuracywhile balancing costs, available resources,and other practical considerations.So let's take a look at it.What you're looking at now is the total survey errorframework.If you look at the bottom there, what you seeis something called a survey statistic.

  • 08:42

    LILIAN YAHNG [continued]: That's the end goal-- an estimate of the population.In all the square boxes are stepsalong the way to getting there.And the circles are sources of potential error.We won't go over everything in detail on this chart,so you should hunt down this resource on your own.But we'll cover some important elementsin a little more detail next.

  • 09:07

    LILIAN YAHNG [continued]: Whom are you surveying?That is, who should fill out your survey?Think carefully about this, as it is often a source of errorand problematic data.First, define the target population.That is, describe who it is you're interested in studying.And if you're interested in studying entities,that is for instance, churches or non-profits,you have to decide who it is you want to send your survey to.And be specific.

  • 09:28

    LILIAN YAHNG [continued]: So an example of a target populationmight run like this-- adults under 30 years of agewho currently ride a bicycle five hours or more in Portland,Oregon.Now you have to consider Portland,Oregon is a very different place from, say, Savannah, Georgia.And also the crew of people who ride their bicyclesonly on holidays or weekends may be different too, as also with

  • 09:48

    LILIAN YAHNG [continued]: age.The lesson there is be specific.Next, build a sampling frame and outline a sampling strategy.How are you going to get at this group of people, your targetpopulation of cyclists?Maybe you might try the city bicycle registration records,or a local bicycle club, or maybe evena random sample of households or residences.

  • 10:09

    LILIAN YAHNG [continued]: And then you might have some eligibility conditionsof age and bicycle time hours.But in any case, be sure to think about howto minimize coverage error.That is, is anyone excluded in your sampling frame?Now if you need to make claims thatare representative to a general population,I would recommend stratified sampling.

  • 10:29

    LILIAN YAHNG [continued]: Consider that this, that you're seeing on screen,is your population.What stratified sampling does is that it divides your populationinto these groups called strata.You might divide your groups by a variety of variables.Some common ones are demographic variables, like genderor ethnicity or income, or other things like that.

  • 10:52

    LILIAN YAHNG [continued]: Once you've divided your strata, what you want to dois use a simple random sampling strategy, or maybesystematic random sampling.So what you do is you take your strata,and then you run a random sample up within that strata.So one way we might do it is just do it as a random one.And another way you might do it is take a systematic sample.So you might count every third or every fourth person.

  • 11:16

    LILIAN YAHNG [continued]: When you are doing this, you need to think, too,about sampling error.Sampling error is the variability in estimatesif multiple samples were drawn.And this is reduced if the sample size is larger,because it would be closer to the population.So you also need to consider the requisite sample size.The term, sample size, is from statistics.It refers to the number of completed surveys

  • 11:37

    LILIAN YAHNG [continued]: you get at the end of your survey.Now don't confuse it with the size of the population,or the number of people in your to contactlist drawn from your sampling frame.So how many completed surveys will you need?Well how precise do your survey estimates need to be?What margin of error can you live with,and at what confidence level?

  • 11:57

    LILIAN YAHNG [continued]: There are various sample size calculatorsonline to help you get an idea of what numbers you're facing.But remember, the sample size is what you end up with,and you may not have 100% compliancein people taking your survey.For one thing, some people may be unreachable,or your contact information may not be current.Or they simply might choose not to take your survey,

  • 12:19

    LILIAN YAHNG [continued]: and you might find other people aren't eligible for your surveyafter all.Back to the Portland bicyclists--if your sampling frame is the city bike registration list,some on it will be over 30 years of age.That means you're going to have to screen out those peopleif your target population is adults under 30.So factor in both your anticipated response rates,

  • 12:40

    LILIAN YAHNG [continued]: and any eligibility conditions when you do your sampling.In other words, you will probablyneed to send your survey to more people--maybe a lot more than your needed sample size.For more information about possible measurement errordue to problems in your questionnaire,

  • 13:01

    LILIAN YAHNG [continued]: be sure to watch the video about questionnaire design.But right now, let's talk about the datayou get from your survey.People filled out your survey, and now youhave a perfect data set, right?Well, not quite.Get to know your data in all of its glorious raw messiness.And here are just a few things to consider-- one,

  • 13:22

    LILIAN YAHNG [continued]: interviewer related issues.If your survey is administered by a person,either in person or over the phone, or even through Skype,you need to consider there may be some variancebetween your data collectors.That is, your interviewers, that may show up in your data.Here you need to address issues of providing consistent,standardized training.

  • 13:43

    LILIAN YAHNG [continued]: And that goes for everyone who codes your text data too.And in fact, any survey administeredby a team of people.In other words, at the data processing stage.The error here would be processing error.That is, if what the coder codes does not trulyrepresent the response that the respondent gave.If you have a paper survey, this could include scanning errors.

  • 14:06

    LILIAN YAHNG [continued]: You will also likely have the challenge of missing dataand what to do with it.Data could be missing in a couple of different ways.One, not everyone in your sample completes your survey.So you might have missing cases.And in some cases, this can lead to non-response error.And two, even if someone does your survey,they might skip some questions on it.

  • 14:27

    LILIAN YAHNG [continued]: You need to decide how you're going to deal with that.And also what to do if a respondent gives implausibleresponses like, that they ride their bike for 30 hours a day.And while you are pouring over your missing data,just remember all the surveys you never did.You were missing data in someone's data set.Let's consider what you will do with open-ended items,

  • 14:49

    LILIAN YAHNG [continued]: if you have any.When you code them, do you have a schemeof key themes or something like that to guide the coding?Also, if who responded to your surveydoes not map very well to your strata or other populationcharacteristics, you may need to weight your dataor perform other post-survey adjustmentsto account for non-response and potentially skewed estimates.

  • 15:15

    LILIAN YAHNG [continued]: The bad news is probably not everyonedid, or will do, your survey.The good news is that doesn't necessarilymean your data will be biased.A response rate, fundamentally, is the percentageof people who took a survey.So it's your survey completions over your total in your sample,minus anyone who you found to be ineligible.

  • 15:38

    LILIAN YAHNG [continued]: Be sure to read more about that and different kinds of responserates at rates are important and it'simportant to report them in your results.Sometimes they're even seen as an indicator of survey quality.But in fact, you may be surprised to note thatby themselves alone, response ratesare not a reliable indicator of survey quality.

  • 15:59

    LILIAN YAHNG [continued]: So don't automatically despair if you have a low responserate.Maybe the non-response was at random.Your data may not be biased.On the other hand, don't automaticallyassume you're safe because your response rate is high.Maybe you left out an important sub-group or contingentin your initial sampling frame.However, high response rates couldlower the risk of non-response bias in your survey data.

  • 16:22

    LILIAN YAHNG [continued]: So some tips to look out for-- compare sub-group breakdownsof survey respondents to your total sample.Take a look at your strata and alsobasic demographic variables, like gender or ethnicityand race.How do they compare?Fundamentally compare survey respondentsand non-respondents.Try to get a sense for who is missing.

  • 16:42

    LILIAN YAHNG [continued]: In our bicycling example, is it mainlymen between the ages of 18 and 24, perhaps?Finally, look to see if you can comparesurvey estimates with other available data sources.Sometimes an external source independentof your survey data, perhaps records or a dataset from a different survey.Do your estimates jive with those external sources?

  • 17:12

    LILIAN YAHNG [continued]: So in conclusion, avoid these naive views.One, anything goes in a survey.And two, any data is good data.As you have seen, neither of those views is true.There is a methodology to the design and implementationof surveys, as well as the analysis of resulting surveydata.

  • 17:33

    LILIAN YAHNG [continued]: And we've only touched on a few aspects of survey methodology.It's just the tip of a very cool iceberg.Seek out and learn more about the total survey errorframework that describes potential sources of errorin sample surveys.And finally, don't forget that youare a participant in the construction of data,

  • 17:53

    LILIAN YAHNG [continued]: whether or not you respond to surveys.And I hope that you do.

Video Info

Publisher: SAGE Publications Ltd

Publication Year: 2017

Video Type:Tutorial

Methods: Survey research, Sampling, Populations

Keywords: comparison; compliance; exclusion; objectives setting; practices, strategies, and tools

Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:



Methodologist Lilian Yahng presents an overview of survey research and design. She describes the types of data that surveys can effectively collect, highlights the steps in survey research, and introduces the total research error method for quality control.

Looks like you do not have access to this content.

An Introduction to Survey Research

Methodologist Lilian Yahng presents an overview of survey research and design. She describes the types of data that surveys can effectively collect, highlights the steps in survey research, and introduces the total research error method for quality control.

Copy and paste the following HTML into your website