Central Limit Theorem

History of the Central Limit Theorem

The term central limit theorem most likely traces back to Georg Pólya. As he recapitulated at the beginning of an article published in 1920, it was “generally known that the appearance of the Gaussian probability density exp (–x2)” in a great many situations “can be explained by one and the same limit theorem” which plays “a central role in probability theory.” Pierre-Simon Laplace had discovered the essentials of this fundamental theorem in 1810, and with the designation central limit theorem of probability theory, which was even emphasized in the article’s title, Pólya gave it the name that has been in general use ever since.

In this article of 1820, Laplace starts by proving the central limit theorem for some certain probability distributions. He then continues with arbitrary discrete and continuous distributions. But a more general (and rigorous) proof should be attributed to Siméon Denis Poisson. He also [Page 255]intuited that a weaker version could easily be derived. As for Laplace, for Poisson the main purpose of that central limit theorem was to be a tool in calculations, not so much to be a mathematical theorem in itself. Therefore, neither Laplace nor Poisson explicitly formulate any conditions for the theorem to hold. The mathematical formulation of the theorem is attributed to the St. Petersburg School of probability, from 1870 until 1910, with Pafnuty Chebyshev, Andrey Markov, and Aleksandr Liapounov.

Mathematical Formulation

Let X1,X2,…,Xn be independent random variables that are identically distributed, with mean μ and finite variance σ2. Let

$${\overline{X}}_n=\frac{X_1+\dots +X_n}{n}$$

denote the empirical average, then from the law of large numbers $[{\overline{X}}_n-\mu \text{]}$ tends to 0 as n tends to infinity. The central limit theorem establishes that the distribution of $\sqrt{n}[{\overline{X}}_n-\mu \text{]}$ tends to a centered normal distribution when n goes to infinity. More specifically,

$$p\left(\sqrt{n}\frac{[{\overline{X}}_n-\mu \text{]}}{\sigma }\le x\right)\to \Phi (x)={\displaystyle \underset{-\infty }{\overset{x}{\int }}\frac{1}{\sqrt{2\text{π}}}exp\left(-\frac{z^2}{2}\right)}dz.$$

We can also write

$$\sqrt{n}\left(\frac{[{\overline{X}}_n-\mu \text{]}}{\sigma }\right)\stackrel{L}{\to }N(0,1)$$

or $\sqrt{n}({\overline{X}}_n-\mu )\stackrel{L}{\to }N(0,{\sigma }^2)$ as n → ∞.

A Limiting Result as an Approximation

This central limit theorem is used to approximate distributions derived from summing, or averaging, identical random variables.

Consider for instance a course where 7 students out of 8 pass. What is the probability that (at least) 4 failed in a class of 25 students. Let X be the dichotomous variable that describes failure: 1 if the student failed and 0 if the student passed. That random variable has a Bernoulli distribution with parameter p = 1/8 (with mean 1/8 and variance 7/64). Consequently, if students’ grades are independent, the sum Sn = X1 +…+ Xn follows a binomial distribution, with mean np and variance P (1 − p), which can be approximated, by the central limit theorem, by a normal distribution with mean np and variance np(1 − p). Here, μ = 3.125 while σ2 = 2.734. To compute P(Sn ≤ 4), we can use the cumulative probabilities of either the binomial distribution or the Gaussian approximation. In the first case, the probability is 80.47%,

$$\begin{array}{c}{\left(\frac{7}{8}\right)}^{25}+25{\left(\frac{7}{8}\right)}^{24}\left(\frac{1}{8}\right)\\ +\frac{25\cdot 24}{2}{\left(\frac{7}{8}\right)}^{23}{\left(\frac{1}{8}\right)}^2+\\ \frac{25\cdot 24\cdot 23}{2\cdot 3}{\left(\frac{7}{8}\right)}^{22}{\left(\frac{1}{8}\right)}^3\\ +\frac{25\cdot 24\cdot 23\cdot 22}{2\cdot 3\cdot 4}{\left(\frac{7}{8}\right)}^{21}{\left(\frac{1}{8}\right)}^4\end{array}$$

In the second case, use a continuity correction and compute the probability that Sn is less than 4 + 1/2. From the central limit

...