Bayes Factors

The problem of comparing multiple competing models is a difficult one and has been the focus of much research over the years. For statisticians of a Bayesian inclination, the Bayes factor offers an easy way of comparing two models. Simply defined, the Bayes factor is the ratio of the posterior and the prior odds. More specifically, denote the data as x = x1, x2,…, xn and the two models as M1 and M2, respectively. The Bayes factor in favor of model M1, and thus against model M2, is

where p(Mi) is the prior probability assigned to model i and p(Mi | x) is the posterior probability of model i after observing the data.

Since the posterior probability of model i can be expressed as p(Mi | x) = p(x | Mi) p(Mi), the Bayes factor is also sometimes given as B12(x) = p(x | M1)/ p(x | M2). It follows that the posterior odds are equal to the Bayes factor times the prior odds.

The first representation provides an intuitive explanation of what the Bayes factor measures, namely, how the data have affected the odds in favor of model M1. If B12(x) > 1, then the posterior odds in favor of the first model are higher than the prior odds in favor of the first model. In other words, the data have increased our relative amount of belief in the first model. If, on the other hand, B12(x) < 1, the posterior odds in favor of model M1 have decreased on observing the data.

In practice, values of B12(x) smaller than 3 are often taken as providing no evidence in favor of M1 over M2, values of B12(x) between 3 and 20 give positive evidence, values between 20 and 150 are indicative of strong evidence, and any value of the Bayes factor over 150 is taken to be very strong evidence. Although these values are only guidelines, they are useful for the calibration of results.

One advantage of Bayes factors over traditional approaches to model comparison is that the models do not have to be nested, as this simple example demonstrates. Suppose we have data x1,…, xn, independent, identically distributed, coming from either a negative binomial distribution with probability p of success (this is model M1) or from a Poisson distribution with mean λ (this is model M2). We use different notation for the parameters of these two models to emphasize that there is no need for the models under consideration to be related to each other in any way. Both these models are completely specified, and so the Bayes factor for comparing the two models is simply the likelihood ratio, that is

Now, suppose that p and λ are not known. To carry out a Bayesian analysis, it is necessary to assign prior distributions to the unknown parameters of the two models. For simplicity, suppose that p ∼ Beta(α1, β1) and λ∼ Gamma(α2, β2). Then it can be shown that

and

the Bayes factor is the ratio of these two. In this case, we need to have information on α1, β1, α2, and β2 in order to evaluate the Bayes factor, which will always depend on the prior specification. As we assign different values to these four parameters, resulting in different prior distributions (representing our different opinions about p and λ and how sure we are of those opinions), the Bayes factor will in turn vary.

...