Error

Modeling

For practical purposes, the universe is stochastic. For example, any “true” model involving gravity would require, at least, a parameter for every particle in the universe. One application of statistics is to quantify uncertainty. Stochastic or probabilistic models approximate relationships within some locality that contains uncertainty. That is, by holding some variables constant and constraining others, a model can express the major relationships of interest within that locality and amid an acceptable amount of uncertainty. For example, a model describing the orbit of a comet around the sun might contain parameters corresponding to the large bodies in the solar system and account for all remaining gravitational pulls with an error term.

Model equations employ error terms to represent uncertainty or the negligible contributions. Error terms are often additive or multiplicative placeholders, and models can have multiple error terms.

Development of Regression

The traditional modeling problem is to solve a set of inconsistent equations—characterized by the presence of more equations than unknowns. Early researchers cut their teeth on estimating physical relationships in astronomy and geodesy—the study of the size and shape of the earth—expressed by a set of k inconsistent linear equations of the following form:

where the xs and ys are measured values and the p + 1βs are the unknowns.

Beginning in antiquity, these problems were solved by techniques that reduced the number of equations to match the number of unknowns. In 1750, Johann Tobias Mayer assembled his observations of the moon's librations into a set of inconsistent equations. He was able to solve for the unknowns by grouping equations and setting their sums equal to zero—an early step toward Σ∊i = 0. In 1760, Roger Boscovich began solving inconsistent equations by minimizing the sum of the absolute errors (Σ|∊i|=0) subject to an adding-up constraint; by 1786, Pierre-Simon Laplace minimized the largest absolute error; and later, Adrien-Marie Legendre and Carl Friedrich Gauss began minimizing the sum of the squared error terms (Sigma;∊2i = 0) or the least squares. Sir Francis Galton, Karl Pearson, and George Udny Yule took the remaining steps in building least squares regression, which combines two concepts involving errors: Estimate the coefficients by minimizing Sigma;∊2i = 0, and assume that . This progression of statistical innovations has culminated in a family of regression techniques incorporating a variety of estimators and error assumptions.

...