[MUSIC PLAYING][An Introduction to Survey Weighting]
DR. TRENT BUSKIRK: Hi, my name is Dr. Trent Buskirk.I am the Vice President of Statistics and Methodologyat the Marketing Systems Group.I do research in data collection with smartphones,survey designs, dual frames, and usingmachine learning to do designs and weighting for surveysand samples.Today I'm going to talk to you a little bit more
DR. TRENT BUSKIRK [continued]: about sample weighting.[Learning Objectives]So I'd like to talk about what the learning objectives arefor today's film.Today we're gonna talk about what sample weights are,we're gonna define what they are, how to calculate them.We'll also talk about the steps that youwould use to adjust sample weights from base weightto final weight.
DR. TRENT BUSKIRK [continued]: And, we'll also give a motivating examplefor why you should care about using base weights for sampledesigns.[What is sample weighting?]So sample weights are basically the fundamental ingredientthat you need to compensate for a design that collectsdata using sample surveys.
DR. TRENT BUSKIRK [continued]: They're directly connected to the design itself.So for example, if you collect data from multiple strata,maybe you want to collect data from men and from women.And you sample men at a higher ratethan you sample women, because theytend to not respond at higher rates than women.Then you might need to compensatefor the different composition in your sample,that you might end up with more men than women
DR. TRENT BUSKIRK [continued]: in the population.And so sample weighting allows you to make that adjustment,prior to analyzing any data from that survey data collectionenterprise.There are several steps that one might incurwhen doing sample weighting.And specifically, the base weight calculationis directly tied to the sample design itself.
DR. TRENT BUSKIRK [continued]: So if you had a stratified sample design, as opposedto a cluster sample design, your approachto computing the base weights, whichis the fundamental connection to the selectionprobabilities for that probability based design,would differ.Simply put, base weights are the inverseof the selection probability of a given unit for a given
DR. TRENT BUSKIRK [continued]: sampling design.So if you're selecting clusters of households,and then selecting people within a household,you would need to have two stages of weightingbecause there are actually two stages of randomization,or selection.The first stage would select the households.So the unit of selection is the household, itself,
DR. TRENT BUSKIRK [continued]: and the probability of selecting a household wouldbe based on the number of households that are available,and the number of households that you desire.Once you select those households,you then want to select an adult within the household.So the selection probability for the second stageof the enterprise would include the numberof adults in the household, and the number
DR. TRENT BUSKIRK [continued]: you wanted to select.Once you have computed those two selection probabilities,you multiply them together, because the selectionoccurred in stages.Once you have the final selection probability,you take the mathematical reciprocal, oneover the selection probability.That becomes your sampling weight, whatwe like to call base weights.
DR. TRENT BUSKIRK [continued]: The base weights are not the end, but the beginning,of the process.Imagine creating a cookie where youwanted to have a chocolate chip cookie with oatmeal.You wouldn't stop with the chocolate chips,because they're not as good unless you include the oatmeal.So, in some ways, the base weightsare like the base ingredient.You have to tie it to the thing you're trying to make,
DR. TRENT BUSKIRK [continued]: or the design that you're using.But once you get to the base weight calculation ,there are many other adjustments that you have to considerin order to complete the weighting process.[Adjusting Sample Weights]The adjustments include things like, "My framedidn't cover all the men and women in the area I
DR. TRENT BUSKIRK [continued]: wanted to survey."Or "I didn't have a list of all the households in a given areathat I wanted to survey."Or "Not everybody responded to the surveylike I thought they would."Or "Not everybody was eligible in the householdthat I surveyed or got on the phone."There are several adjustments thatneed to be made in order to, A, refine
DR. TRENT BUSKIRK [continued]: the information that you've gotten from the base weightsso that they're applicable to the right populationthat you're trying to infer.And, B, information to account for biases that mighthave crept up along the way.For example, selection bias, nonresponse bias, coveragebias.These are all issues that we needto deal with on the back end of the process that are associated
DR. TRENT BUSKIRK [continued]: with the design, but that can be correctedfor through a series of steps in a sample weighting process.So I want to talk a little bit now about allof the adjustments that you might make once youhave computed the base weight.But before I do that, I did want to mentionone other thing about base weightsthat are really important.Probability designs and non-probability designs
DR. TRENT BUSKIRK [continued]: differ in a very fundamental piece.The non-probability designs don't allow everyoneto have an a priori or known probability of selection that'sgreater than zero.Probability designs, though, by their very nature,give everyone a selection probabilitythat is greater than zero.It is this property of probability designsthat allows us to create a base weight that
DR. TRENT BUSKIRK [continued]: could be very different for everybody in the sample.And we can create a non-zero inclusion probability,or selection probability, for everyone in the sample.This is an extremely important ideato remember when you're working with probability based samples.The other thing is, not every probability designwill end up with the same base weight or the same selection
DR. TRENT BUSKIRK [continued]: probability.Because you could have things like an equal probabilityof selection design like a simple random samplewhere everybody is included at the same chance.In that situation, the weighting is constant, it's very easy.Everybody gets the same weight.But in other designs, where you might say,you're going to oversample Hispanics or undersample
DR. TRENT BUSKIRK [continued]: other minorities, or oversample women for a given study,then you can imagine that the base weights or the inclusionprobabilities would be different for those peoplethat you oversampled compared to those peoplethat you undersampled.And so it is this property that the sample weightingbecomes interesting.Because now you have a different selection probabilityfor different people, or different strata
DR. TRENT BUSKIRK [continued]: in your design.This is not to say that this is not a random sample,it is a random sample with different selectionprobabilities that need to be accounted for.And it is this particular aspect of sample designs,and if not accounted for in the base weights,could lead to bias in your estimates.Bias that is not associated with other non-sampling mechanisms,
DR. TRENT BUSKIRK [continued]: but bias that's associated with the sampling mechanism, itself.Let me give you an example.If you oversampled sampled Hispanics and asked people,how often do you eat Hispanic food?You might imagine that if you don't accountfor the oversampling of Hispanics,you might have greater representation or greaternumbers of meals that are reported that areof ethnic or Hispanic food.
DR. TRENT BUSKIRK [continued]: But if you were to properly accountfor the oversample of Hispanics using the properly accountedfor base weights, you would bring those estimates backin line and it would be more on target,or less biased if you will.So the base weighting is really important for thinkingabout things in the design that might have over representedor under represented folks as a result of your selection
DR. TRENT BUSKIRK [continued]: mechanisms, not the other non-sampling mechanismsthat we'll talk about in a moment.Once you've computed the base weightcalculations for everyone in the sample,and it's tied to the sample design,you must remember the base weights,in order to start with the base weight calculations,you have to understand what the design is.That is the blueprint for which you would derive
DR. TRENT BUSKIRK [continued]: these selection probabilities.But once you have these selection probabilities,the base weights form the fundamental ingredientof the further select stages of adjustment.Those stages of adjustment include: frame integration,multiplicity adjustment, eligibilityor unknown eligibility adjustment,
DR. TRENT BUSKIRK [continued]: nonresponse adjustment, and calibration adjustments.I'm gonna talk about each of these,in turn, because I think they're reallyimportant parts of the overall weighting process.But before I talk about them, I didwant to mention one thing that's important.And that is not every survey willbe amenable to using all possible adjustments.
DR. TRENT BUSKIRK [continued]: For example, if you used an address-based sampling designwhere you want any household to respond to your survey,regardless of who is in that household,then there would be no eligibility or unknowneligibility adjustment because everyone is eligible.Good for you, it's one less stage to worry about.But if you had a telephone surveywhere you were worried about the age of the person
DR. TRENT BUSKIRK [continued]: being 18 or older, and you've got a chatty teenager whois 14 on the phone, you would certainlyimagine they weren't eligible for that survey.So you might want to make an eligibility adjustmentin that case.So not all of these processes or adjustmentswill be applicable to all surveysbut they are important to consider in the process
DR. TRENT BUSKIRK [continued]: overall.[Frame Integration Adjustment]If you have a survey design or a sampling designthat incorporates multiple sampling framesto cover the population that you care about.For example, if you survey the telephone population in the US,you will likely include elements of the sample
DR. TRENT BUSKIRK [continued]: from the landline side and from the cellphone sideto cover all possible telephone users in the US.You would likely need to incorporate the frameintegration adjustment, but simply adjust for the factthat people who have both a landline telephoneand a cellphone could've been included in your sampleat higher rates than those people who
DR. TRENT BUSKIRK [continued]: had either just a land line phone or just a cellular phone.So the frame integration approachessentially allows for an adjustmentfor the multiple ways that peoplecould be included in your sample design,by virtue of the fact that they could be on one or moreof the frames that you're using to cover that targetpopulation.[Multiplicity Adjustment]
DR. TRENT BUSKIRK [continued]: A similar adjustment technique is calledthe multiplicity adjustment.If you surveyed someone with a landline telephoneand they told you they have three landline telephonesfor which you could've reached them for this survey,you would want to incorporate that adjustment into the sampledesign weighting process in order to account for the factthat this person had three ways to be selected for your survey.
DR. TRENT BUSKIRK [continued]: And, in fact, you only want to use them once.So the multiplicity adjustment is very relatedto the frame integration adjustment,but it is specifically used for many multiple ways people couldbe contacted on a given frame.[Unknown Eligibility Adjustment]
DR. TRENT BUSKIRK [continued]: The unknown eligibility adjustmentis one that would require you to understand whether or notsomeone that you surveyed is going to beeligible for the survey itself.If for example, you wanted to survey adults 18 and older,and you had a screener for age, and the person did notanswer or tell you how old they were,
DR. TRENT BUSKIRK [continued]: then their eligibility might be unknown.Because you don't have information on their ageto verify the eligibility.So that person would be an unknown respondentor a respondent of unknown eligibility.So you would want to make accountfor that in the weighting processby ensuring that those who were eligible would represent,not only themselves, but those people
DR. TRENT BUSKIRK [continued]: who had unknown eligibility.[Nonresponse Adjustment]Nonresponse adjustments are necessary to incorporateinformation from respondents and nonrespondentsin order to account for nonresponse bias thatmight be possible, because people justdecided not to participate in your survey for a myriad
DR. TRENT BUSKIRK [continued]: of reasons.With survey response rates dropping,it is really important to at least understandthe possibility that a low response ratesurvey might have with it a corresponding potentialfor bias.And the bias is related to the factthat you just did not get everyone in your sample
DR. TRENT BUSKIRK [continued]: that you tried to get for whatever reason.So this goes beyond unknown eligibility.These people are considered to be eligible,but they did not respond to your survey.They did not want to participate,they were not available to participate, for whateverreason they chose not to.In this situation, you have respondents who participated
DR. TRENT BUSKIRK [continued]: and nonrespondents who did not participate.No data is available for the nonrespondents for the outcomesthat you care about, but you might have frame dataon those nonrespondents.For example, in address-based samples,for the nonrespondents, you mightbe able to append census-related informationor household-specific information thatis available from the sampling frame or from a sample vendor
DR. TRENT BUSKIRK [continued]: to understand more information about nonrespondentsand respondents.That information, together, can beused to make an adjustment for the nonresponse.In the simplest case, the nonrespondentsare grouped with like respondents,and the sampling weights of the like respondentsare increased to compensate for the nonrespondents who didn't
DR. TRENT BUSKIRK [continued]: come along for the party.One additional point I wanted to clarifyabout the nonresponse adjustments,in particular, is that sometimes nonresponse adjustments,while attractive to compensate for the potentialof nonresponse bias, are a bit difficult to carry outin practice simply because of the limited informationthat's available.In order for these adjustments to work,you have to have information from both the respondents
DR. TRENT BUSKIRK [continued]: and the nonrespondents.Sometimes, sample designs are notamenable for this information.For example, if you do a telephone surveyand you have respondents who answer the phoneand participate in your survey and give you lots of cool data,and you have nonrespondents that hang up and tell youthey're not interested in taking the survey,you have very little information on those nonrespondents.
DR. TRENT BUSKIRK [continued]: You have their telephone number and maybe youmight be able to locate their geography by virtueof the area code and prefix.But beyond that, there's very little informationthat you have on those nonrespondents.So it's unlikely that a formal nonresponse adjustmentstep would be actually useful, or createmuch utility in the sample weighting process.In that case, you're more likely to use weight calibration,
DR. TRENT BUSKIRK [continued]: or calibration or post stratification adjustment.[Calibration Adjustments]Calibration and post stratification adjustmenttechniques basically take the informationthat you have available from respondents,like demographic variables and other related socioeconomicvariables for which you have external population
DR. TRENT BUSKIRK [continued]: controls for, and you basically calibrate the baseweights of the respondents using those variablesto the population control totals.So in this way, you might say that the sampleis representative, or looks, more like the United Statespopulation in terms of the demographics.
DR. TRENT BUSKIRK [continued]: The weighted distribution of men and womenis the same as the census, or is the sameas your external control total.The distribution of race ethnicityis the same as your external control totals, and so on.In this way, the calibration processattempts to mitigate issues that are relatedto nonresponse bias, that are related to coverage bias,
DR. TRENT BUSKIRK [continued]: by excluding certain households or excludingcertain demographic groups or not including them at the ratesthat you might expect to include from census countsor other distributions that are external.Calibration is usually most always usedin the final process for additional reasons,like unit conversion.
DR. TRENT BUSKIRK [continued]: Let me give you an example.If you select telephone numbers in your sample of the USpopulation, the unit of selectionis the telephone number, not the person.So when you're talking about the base weights, the baseweights for your sample without nonresponseor any other consideration, would sum upto the universe of telephone numbers, which
DR. TRENT BUSKIRK [continued]: is a lot bigger than the universe of the United States.And so, in that way, you need to calibrate your base weightsto the population that you're actuallytrying to make inference for.In this case, the human US adult population.So calibration can also serve as a unit change, if you will,moving from the telephone unit to the adult unit.
DR. TRENT BUSKIRK [continued]: That's another role the calibration can play.Calibration doesn't require informationfrom nonrespondents, and it can account for coverage issuesand other issues like that.The effectiveness of the calibration techniques,though, are tied to the correlation between,or the association between, the variables you'reusing in the calibration models and the survey outcomes
DR. TRENT BUSKIRK [continued]: that you care about, as well as the response mechanism.So if you realize that males respond less oftenthan females, and that being maleis somehow associated with your survey outcome,then you certainly would want to include genderas a calibration variable.Sometimes, you can include calibration variableson their own: males, race, ethnicity, income status,
DR. TRENT BUSKIRK [continued]: education status.But sometimes you get a bigger effectif you can include the cross tabulation,or the cross classification, of those calibration variables.So for example, you might consider female Hispanic, maleHispanic, female African American, male AfricanAmerican, and you're calibrating to a cross tabulation
DR. TRENT BUSKIRK [continued]: of variables, as opposed to just a single variable.This is especially important if youthink that there could be a relationship between gender,ethnicity, and response or gender, ethnicity,and your outcome variables, in particular.So it's something to consider whenyou're thinking about moving forward with calibration step.And the calibration step is generally
DR. TRENT BUSKIRK [continued]: the final step of the sample weighting process.[What information od you need to do these adjustments?]So I want to talk a little bit now about the informationthat you will need to incorporatein each of the adjustment phases that we talked about recently.If you are doing a multiplicity adjustmentor an unknown eligibility adjustment,
DR. TRENT BUSKIRK [continued]: and you have information availablefor your entire sample, then you willwant to include the entire samplein that set of adjustments.If you want to perform a nonresponse adjustment,and you have information available for respondentsand nonrespondents, the nonrespondents and respondents
DR. TRENT BUSKIRK [continued]: are the information is needed for just those folksto perform the nonresponse adjustment.So let me clarify.If you have folks that have unknown eligibilityin your sample, and you make an adjustmentfor unknown eligibility, and thenyou move to the nonresponse adjustment phase,
DR. TRENT BUSKIRK [continued]: all the folks that were of unknown eligibilityare now expunged from the data fileand are not included in the nonresponse adjustmentbecause you've already accounted for that in the weightingprocess.Moving to the nonresponse adjustment,you will take respondents and nonrespondents,and all the information you have available for them,
DR. TRENT BUSKIRK [continued]: and use a model, like a propensity score adjustmentor a stratification adjustment or modelingadjustment or waiting class adjustment,to adjust the respondents and the nonrespondents together.Once you have created the response or nonresponseadjustment weight portion of the process,
DR. TRENT BUSKIRK [continued]: the nonrespondents are then removed from the fileand calibration will continue with just the respondents.So you just have to keep in mind that along the way,there are several steps for the weighting process,but those steps will use certain pieces of data,or certain types of respondents, as you whittle down and make
DR. TRENT BUSKIRK [continued]: the final adjustments.And, of course, once all the adjustments are done,you would include the respondents, all the sampledata you collected, and the final weight that is basicallythe product of all of the adjustments from the baseweight moving forward for all the stagesthat you incorporated in the weighting adjustment process.[Why is sample weighting necessary?]
DR. TRENT BUSKIRK [continued]: I now want to talk a little bit about why sample weightingmight even be necessary.We talked a little bit about what sample weighting is,and how it might be accomplished,through several stages of adjustment from the baseweight moving forward.But I did want to motivate why base weighting, or sampleweighting, is needed in general.
DR. TRENT BUSKIRK [continued]: And as we mentioned earlier, sample weightsallow you to compensate for the design, the probabilityof selection, and potential aspects of the design thatcould be related to your eventual estimatesand your eventual inference.I'm gonna motivate this example with two strata.Suppose that you wanted to know about how many hours people
DR. TRENT BUSKIRK [continued]: watch TV on a typical week.You're interested in advertising or othersome aspects that are related to television consumption in mediaviewing.So you actually think that owners might watch televisionless often than people who rent.Or, maybe, it's vice versa.But you have a working hypothesis
DR. TRENT BUSKIRK [continued]: that would differentiate the number of television viewinghours you might expect by people's owner/renter status.So you decide to stratify your sample by ownership status,so you have a stratum of owners and youhave a stratum of renters.In this example, we have a populationof ten owners and six renters.
DR. TRENT BUSKIRK [continued]: Because owners are a little harder to reach, and renters,by virtue of the places that they live,we will select two owners from the population of ten.And we will select three renters from the population of six.So the selected owners are indicated here,along with the selected renters.
DR. TRENT BUSKIRK [continued]: If we consider our sample altogether,we have the two owners and the three renters.If we take the average televisionviewing hours for the last week, wesee that on average, for this sample,it was about 6.2 hours watched last week.You can see that, remember, that the populationaverage was about 4.5.
DR. TRENT BUSKIRK [continued]: So we have a slight overestimate in our estimatefrom the sample taken together.However, we remember that our hypothesis going into itwas that renters would watch more television on averagethan owners.And in our sample, we have three renters and two owners.And so it's not surprising that wehave a slightly larger estimate compared to the population
DR. TRENT BUSKIRK [continued]: average.But remember, this is the unweighted sample.Let's see what we would get if we weightedthe sample accordingly.Remember that the weights at this particular stagefor this simple example is the inverse of the selectionprobability.We had ten owners and we selected two,so the selection probability for any owner is two over ten,
DR. TRENT BUSKIRK [continued]: or one fifth.The sample weight would just be the reciprocalof that, which would be five.For the renters, we selected three rentersfrom a population of six.So the selection probability would be three over six,or one half.The sample weight would just be the reciprocalof that, which would be two, as we've indicated here.Now, if we compute the weighted average, which is just
DR. TRENT BUSKIRK [continued]: simply multiplying the weight times the y variable,or the outcome the number of television hourswatched, we add all of those up and thenwe divide by the sum of all the weights, which in this casewould be the size of our population, or 16,we get a sample average of about 4.8.Now the sample average is never going to exactly equal
DR. TRENT BUSKIRK [continued]: the population average.But on average, that would happenthrough repeated sampling, that your sample average wouldbe exactly what the population average would be,presuming that there's no nonresponse and anythingelse that might occur.But the theory, at least, says that you might get it righton average on the long haul.And so you'd expect that the sample weight,
DR. TRENT BUSKIRK [continued]: in this situation, would be closerto the unweighted version of that estimate.And in fact, it is.We're making sure that the renters arebeing attenuated in the right proportionrelative to the owners.Remember, in our sample, we had more rentersthen we had in the population.We have six renters in the populationand ten owners in the population.
DR. TRENT BUSKIRK [continued]: But in our sample, there were more renters.So we compensated for that using the base weight calculations,and then we saw that our sample average was, in fact, closerto the population average, which iswhat we would expect if the sample weighting is appliedcorrectly.In this simple example, we didn't have any nonresponseand we didn't have any unknown eligibility to speak of,so we didn't include any of those adjustments.
DR. TRENT BUSKIRK [continued]: But you can see that the relative impact of the sampleweighting brought us back to the right direction.[Summary]To summarize, I wanted to talk one more timeabout the workflow of the sample weighting process.So, keeping in mind, sample weightsare constructed from the probabilitydesigns for which you actually use to collect the data.
DR. TRENT BUSKIRK [continued]: So in order to calculate the base weight,you need the probability of selection,which is tied to the type of design that you used.The second stage of the sample weighting processis adjustments.We talked about several different adjustmentsthat one could make, including multiplicity adjustments, frameintegration adjustments, adjustmentsfor unknown eligibility, and adjustments for nonresponse,
DR. TRENT BUSKIRK [continued]: and in final, the calibration adjustments.And finally, if your survey requires it,replicate weights may be generated.And if you need replicate weights,they allow you to incorporate, or estimate,the variability of your survey outcome estimatesthat might be achieved or acquired from a sampledesign that is complex.
DR. TRENT BUSKIRK [continued]: Replicate weights essentially repeatthis process using repeated subsamplesof your original sample.In that situation, I would compute the base weights,I would compute the adjustments, and so on for every subsamplegenerate replicate weights, and those replicate weightswould form the basis of my inference.Not every sample or survey will use replicate weights.
DR. TRENT BUSKIRK [continued]: But if you do use them, you have to makesure that you incorporate every stage of the adjustmentsthat we've already talked about that you used to computeyour final sampling weights.[Further Reading][MUSIC PLAYING]
Publisher: SAGE Publications Ltd
Publication Year: 2017
Segment Num.: 1
Professor Trent Buskirk outlines the concepts behind survey weighting. He discusses adjustments that can account for over-representation, under-representation, and even nonresponse.
Looks like you do not have access to this content.
Professor Trent Buskirk outlines the concepts behind survey weighting. He discusses adjustments that can account for over-representation, under-representation, and even nonresponse.