Unbiased Estimator

Let the data be represented by X1, X2,…, Xn, a collection of random variables, whose behavior can be modeled by a probability distribution Fθ, where θ represents a parameter to be estimated from the data. Here, θ may be a k-dimensional vector. For example, Fθ may be the family of normal distributions with θ representing the mean of the Xis, or θ may be a two-dimensional vector, θ = (μ, σ2), representing the mean and variance of the Xis. Let δ(X1,…, Xn) be a function of the data. That is, δ(X1,…, Xn) is an estimator.

The estimator δ(X1,…, Xn) is said to be unbiased for g(θ) if, for all θ,

When Condition 1 does not hold, the bias of δ is defined as b(θ) = Eθ(δ(X1,…, Xn)) − g(θ). Gauss introduced the concept of unbiasedness to denote lack of systematic error in the estimation process.

Example 1. Let X1,…, Xn denote random variables with the same expectation θ. Because E(Σni=1ci Xi) = θ, whenever Σni=1ci = 1 then (Σni=1ciXi) is unbaised for θ.

Example 2 (Survey Sampling). Let a population consist of N individuals, each with annual income ti, i = 1,…, N. To estimate the total income T = Σni=1ti take a random sample of size n and denote by {y1, …, yn} the incomes of the n sampled individuals. Note that each of the {y1, …, yn} has probability 1/N of being equal to each of the ti, i = 1,…, N.

The following calculation shows that N/nΣni=1Yi is unbiased for T:

In most cases, the requirement of unbiasedness as defined by Condition 1 yields intuitive estimators with good properties. However, unbiased estimators may not exist, and when they do exist, sometimes they behave poorly.

Example 3. Let X be a binomial random variable with parameters n and θ = probability of success. It is known that there is no unbiased estimator for g(θ) unless g(θ) is a polynomial in θ of degree less than or equal to n. For example, there is no unbiased estimator for 1_θ based on X. On the other hand, when a[Page 1026]geometric random variable Y with θ = probability of success is observed, Y is unbiased for 1/θ.

Example 4. Let X be a Poisson random variable with parameter θ. When estimating (p{X = 0})2 = e−2θ, the unique unbiased estimator is δ (X) = (-1)X, a silly estimator. Lehmann considers this issue and proposes that it is due to inadequate information. For example, if, instead of a single Poisson observation, there are n independent Poisson observations, setting T = Σni=1Xi an unbiased estimator for e−2θ is δ (T) = (1−2/n)T, which is a reasonable estimator for n > 2.

A general concept of unbiasedness was proposed by Lehmann. Let (L(δ(X), g(θ)) represent the loss incurred in estimating g(θ) by δ(X). Then δ(X) is said to be L-unbiased with respect to the loss function L if, for all θ∗ ≠ θ,

When the loss function is squared-error loss, L(δ(X), g(θ)) = (δ(X) − g(θ))2, Condition 2 is equivalent to Condition 1. When the loss function is absolute error, L(δ(X), g(θ)) =|δ(X) − g(θ)|, Condition 2 is equivalent to median-unbiasedness; that is, Condition 2 is equivalent to requiring that δ (X) satisfy the condition that for all θ,