Posterior Distribution

In Bayesian analysis, the posterior distribution, or posterior, is the distribution of a set of unknown parameters, latent variables, or otherwise missing variables of interest, conditional on the current data. The posterior distribution uses the current data to update previous knowledge, called a prior, about that parameter. A posterior distribution, p(θ|x), is derived using Bayes’s theorem

$$p\left(\theta |x\right)=\frac{p\left(x|\theta \right)p\left(\theta \right)}{p\left(x\right)}=\frac{p\left(x|\theta \right)p\left(\theta \right)}{\int p\left(x|\theta \right)p\left(\theta \right)d\theta },$$

where θ is the unknown parameter(s) and x is the current data. The probability of the data given the parameter p(x|θ) is the likelihood L(θ|x). The prior distribution, p(θ), is user specified to represent prior knowledge about the unknown parameter(s). The last piece of Bayes’s theorem, the marginal [Page 1274]distribution of data, p(x), is computed using the likelihood and the prior. The distribution of the posterior is determined by the distributions of the likelihood and the prior and scaled by the marginal distribution of the data. Therefore, the posterior can be represented as

$$\begin{array}{l}\text{Posterior distribution}\propto \text{Likelihood}\\ \times \text{Prior distribution,}\end{array}$$

where ∝ means “proportional to.” The relationship between the posterior, the prior, and the likelihood is shown in Figure 1.

Figure 1 The likelihood and the prior determine the posterior distribution

The prior distribution is conjugate to the likelihood if the resulting posterior distribution has the same form as the prior distribution. The mean and variance of the posterior distribution are also determined by these two distributions. In certain situations, the posterior mean is a weighted average of the mean of the data and the prior, using the precision of each as weight. The precision, the reciprocity of variance, of the posterior is a function of the precision of the data and the prior. Thus, when a researcher is more confident in a prior, it is given more weight by specifying a smaller variance for the prior distribution.

The posterior distribution can be analytically computed by integration or it can be approximated using a Markov chain Monte Carlo algorithm. With increases in computational power, the latter is often the easier option, and the Markov chain Monte Carlo method is what is used in software such as WinBUGS and Mplus. A commonly used Markov chain Monte Carlo method is Gibbs sampling, which recursively generates random numbers from the conditional posterior distribution for each parameter in turn, conditional on the current values of all other parameters.

The resulting posterior distribution is what is used to make inferences about the model. The mean, median, or mode of the posterior distribution can be used as a point estimate, much like a maximum likelihood estimate (MLE) can be used within the frequentist framework. If the prior p(θ) is a constant, the mode of the posterior, if it exists, is equivalent to the MLE. Credible intervals can also be constructed using the posterior distribution. These are analogous to confidence intervals in the frequentist framework but differ in theory and interpretation. Credible intervals provide the (1 − α)% probability that a parameter lies between a lower and upper bound. Thus, credible intervals assume the parameter is random and the lower and upper bounds are fixed, whereas confidence intervals assume the opposite.