Estimation Bias

The Concept ofError in Statistical Inference

Suppose that we would like to estimate a population parameter θ (e.g., population mean). An estimator $\widehat{\theta }$ is any sample statistic (e.g., sample mean) that is used to estimate θ. Because $\widehat{\theta }$ is sample based, it does not perfectly agree with the true value of θ. The difference between the values of the estimator and the parameter, $\widehat{\theta }-\theta$, is called the error of estimation.

In the sampling theory, statistical performance of an estimator is evaluated by the smallness of error in the long run, in which one assumes that an infinite number of random sampling from the population is possible and considers the average size of error in the repeated sampling. For any estimator $\widehat{\theta }$, the following equation generally holds:

$$E{(\widehat{\theta }-\theta )}^2=E{(\widehat{\theta }-E(\widehat{\theta }))}^2+{(E(\widehat{\theta })-\theta \text{)}}^2,$$

where the operator E refers to taking the expectation of the subsequent variable with respect to the sampling distribution of $\widehat{\theta }$. The left-hand side is called the mean squared error (MSE), which is the average squared error of estimation. In general, estimators with smaller MSE are preferred to others when there are competing estimators of the same population parameter. The first term on the right-hand side is the variance of the estimator, indicating the average squared deviation of the estimator from its expected value (i.e., the mean of $\widehat{\theta }$ in the long run). The second term on the right-hand side is the squared difference between the expected value of $\widehat{\theta }$ and the true parameter value. The difference in the parentheses is called the bias of estimator $\widehat{\theta }$ for parameter θ, that is:

$$\text{Bias}(\widehat{\theta },\theta )=E(\widehat{\theta })-\theta \text{.}$$

The equations imply that the total error of an estimator can be decomposed into the variance and bias components. The variance component represents the amount of random error, or precision, which is the unpredictable fluctuation due to sampling and cancels out in the long run. In contrast, the bias component represents the amount of systematic error, or accuracy, which remains constant throughout sampling occasions. In order for the MSE to be small, both variance and bias must also be small.

Implications and Uses in Statistics

The MSE equals the variance of estimator if and only if the bias is zero (i.e., $E(\widehat{\theta }\text{)}=\theta$). Such an estimator is said to be unbiased (it is biased, otherwise). Unbiasedness is considered as one of the desirable properties for estimators. The sample mean, for example, is an unbiased estimator of the population mean if it is calculated from a random sample. Also, maximum likelihood estimators are asymptotically unbiased.

[Page 606]Suppose a random sample of size n (X1, X2, … , Xn) from the normal distribution with mean µ and variance σ2. On the one hand, the sample mean $\overline{X}={\displaystyle {\sum }_{i=1}^n{X}_i/n}$ is an unbiased estimator of the population mean µ because its expected value is $E(\overline{X})=\mu$. On the other hand, the sample variance $S^2={{\displaystyle {\sum }_{i=1}^n(X_i-\overline{X})}}^2/n$ is biased with respect to the population variance σ2 because its expected value is E(S2) = (n − 1)σ2/n. In this case, the amount of bias is Bias (S2,σ2)=E(S2)−σ2=−σ2/n. Thus, the sample variance tends to underestimate the population variance, although the bias diminishes as the sample size n becomes large. In order to obtain an unbiased estimator of the population variance, one can use the unbiased variance ${\widehat{\sigma }}^2={\displaystyle {\sum }_{i=1}^n{(X_i-\overline{X})}^2/(n-1)}$. Yet, it can be shown that the MSE of S2 is smaller than that of ${\widehat{\sigma }}^2$; S2 tends to produce estimates closer to the true population variance than ${\widehat{\sigma }}^2$ in the long run, even though those estimates tend to be smaller than the true variance.

...