Bayesian Statistics

Bayesian statistics is a comprehensive and systematic interpretation of the field of statistics based on the quantification and manipulation of uncertainty in the form of probability distributions enabled by the Bayesian interpretation of probability laws. It is commonly considered a branch of statistics. This entry provides a basic description of Bayesian statistics. Although Bayesian statistics includes nonparametric approaches, the entry’s scope is limited to the more common setting of parametric Bayesian inference.

[Page 171]Although the historical details surrounding the origins of Bayesian statistics are somewhat hazy, the broad strokes are well-documented. The basic theoretical machinery underpinning Bayesian statistics was established in the second half of the 18th century with the discovery of Bayes’s theorem by Thomas Bayes, Richard Price, and, independently, Pierre-Simon Laplace. If beliefs concerning unknown quantities are represented with probability distributions, Bayes’s theorem provides a mechanism to update the beliefs upon the arrival of new evidence, thus laying a foundation for scientific reasoning. This way of thinking became the de facto standard among statisticians from the time of its discovery until well into the 20th century under the name of inverse probability.

The modern term Bayesian statistics grew out of a great and divisive disagreement in the 20th century concerning the interpretation of probability as a contrast to the term frequentist statistics—statistics viewed from the perspective of the frequentist interpretation of probability. Although the frequentist methods of Ronald Fisher, Jerzy Neyman, and Egon Pearson dominated statistical thought for the majority of the 20th century, Bayesian methods did not achieve widespread use until the mid-to-late 20th century due to computational challenges of the paradigm. These challenges were significantly alleviated near the end of the 20th century due to the proliferation of the personal computer and advances in Monte Carlo algorithms.

By the late 20th century, Bayesian ideas had permeated virtually every area of statistical science. Now in the early 21st century, Bayesian methods continue to flourish and expand to larger audiences; however, frequentist methods are still more commonly used in practice and accepted by regulatory agencies. They also comprise virtually all introductory statistics education.