Prior Distribution

A prior distribution p (θ) is a probability distribution describing one's subjective belief about an unknown quantity θ before observing a data set x. BAYESIAN INFERENCE combines the prior distribution with the likelihood of the data to form the POSTERIOR DISTRIBUTION used to make inferences for θ. This entry describes the prior distribution's role in Bayesian inference from four points of view: subjective probability, objective Bayes, penalized likelihood, and hierarchical models.

At its foundation, Bayesian inference is a theory for updating one's subjective belief about θ upon observing x. Subjectivists use probability distributions to formalize an individual's beliefs. Each individual's prior distribution is to be elicited by considering his or her willingness to engage in a series of hypothetical bets about the true value of θ. Models are required to make prior elicitation practical for continuous parameter spaces. Because of their computational convenience, conjugate priors are often used when they are available. A model has a conjugate prior if the prior and posterior distributions belong to the same family. Table 1 lists conjugate likelihood-prior relationships for several members of the exponential family. The prior parameters in many conjugate prior-likelihood families may be thought of as prior data, which provides a straightforward way to measure the strength of the prior (e.g., “one observation's worth of prior information”). See the entry on posterior distribution for an example calculation using conjugate priors.

One often finds that any reasonably weak prior has a negligible effect on the posterior distribution. Yet counterexamples exist, and critics of the Bayesian approach find great difficulty in prior elicitation (Efron, 1986). Objective Bayesians answer this criticism by deriving “reference priors,” which attempt to model prior ignorance in standard situations. Kass and Wasserman (1996) review the extensive literature on reference priors. Many reference priors are improper (i.e., they integrate to ∞). For example, a common “noninformative” prior for mean parameters is the uniform prior p (θ) α 1, which is obviously improper if the parameter space of θ is unbounded. Improper priors pose no difficulty so long as the likelihood is sufficiently well behaved for the posterior distribution to be proper, which usually happens in simple problems in which frequentist procedures perform adequately. However, when improper priors are used in complicated models, the propriety of the posterior distribution can be difficult to check (Hobert & Casella, 1996).

One difficulty with reference priors is that noninformative priors on one scale become informative after a change of variables. For example, a uniform prior on log σ 2 becomes p (σ2) ∝ 1/σ2 because of the Jacobian introduced by the log transformation. Jeffreys' priors are an important family of reference prior that are invariant to changes of variables. The general Jeffreys' prior for a model p (x|θ) is p (θ) ∝ det(J (θ))1/2, where J (θ) is the Fisher information from a single observation.

...