Sampling Error

Any attempt to use a sample survey as a means of studying a larger population of interest runs the risk that the selected sample will not have exactly the same characteristics as the population. In consequence, sample-based estimates will not necessarily take the same numeric values as the population parameters of which they are estimates. The term sampling error is used to refer to this general tendency of sample estimates to differ from the true population values. A main aim of sample design is to minimize the risk of sample characteristics differing greatly from population characteristics.

In the context of quantitative sample surveys using random sampling methods (see random sampling), sampling error also has a more specific meaning. It refers to the expected deviation, under a particular specified sample design, of a particular sample statistic from the equivalent population parameter. This expected deviation will have two components, a component due to random sampling error (variability, or variance) and a component due to systematic sampling error (bias). Therefore, for unbiased sample designs (or, strictly, unbiased sample-based estimators), sampling error is simply equal to sampling variance. Often, standarderrors ofsurveyestimates (the square root of the sampling variance) are estimated and presented as estimates of sampling error. Standard errors can be used in the construction of confidence intervals. More generally, mean squared error is used as a measure of sampling error. Mean squared error is defined as

where the sample statistic y is used to estimate the population parameter Y. The MSE is the sum of the variance and the squared bias of the estimate.

This quantity is, of course, specific to a particular estimate under a particular sample design. In practice, it can never be known for most survey estimates because the magnitude of bias is not known (if it was known, it could be removed from the survey estimate, so there would be no need to use a biased estimator). It is largely for this reason that attempts to estimate sampling error are typically restricted to estimation of the variance of an estimate. Also, there are many situations in which sampling bias can reasonably be assumed to be zero, or negligible. The variance of estimates can be readily estimated from survey data, provided that important design features such as clustering, stratification, and design weighting are indicated on the data and taken into account appropriately in the calculation. For most of the commonly used sample designs, the form of the variance is known; therefore, standard estimation methods can be used. For particularly complex designs, or designs that are not well specified, it may be necessary to use a replication method to estimate empirically the variance of sample statistics. Methods such as jackknife, bootstrap, and balanced repeated replications are increasingly available in commercial software packages.

Sampling error is only one component of total survey error. The total survey error of a statistic is the expected deviation of the statistic from the equivalent population parameter. This incorporates not only error due to sampling, but also error due to noncoverage; [Page 992]nonresponse; and measurement (interviewer error, respondent error, instrument error, processing error). In many situations, although sampling bias may be negligible, other sources of bias may be important. For thisreason, standard errors can be misleading as a guide to the accuracy of survey estimates.

...