Error Term

Error Term Defined

With simple linear regression, a regression model can be expressed as follows:

y = b0 + bx + ε.

[Page 426]In this equation, y is the dependent variable, x is the independent variable, b is the slope of a regression line, and b0 is the line’s intercept on the y-axis. In this particular equation, for the ith observation of the independent variable xi and dependent variable yi, the corresponding error term εi is the difference between the actual value of yi and the value predicted with the equation:

ypredicated = b0 + bxi.

A regression line that fits the population well should have a small amount of error, and in particular, the best fit line in simple linear regression is that which has the smallest sum of all of the error terms squared ($\mathrm{\Sigma }{\mathrm{\epsilon }}_i^2$).

Similarly, all of this can be extended to multiple linear regression with k independent variables with the equation:

$$y=b_0+b_1{x}_1+...+b_k{x}_k+\mathrm{\epsilon }.$$

In this equation, y still represents the dependent variable, x1 through xk represent the k independent variables, b1 through bk represent the corresponding regression coefficient, and b0 represents a constant value. In this particular equation, for the ith observation of the independent variables x1i through xki and dependent variable yi, the corresponding error term εi is the difference between the actual value of yi and the value predicted with the equation:

$$y_{\mathrm{predicted}}=b_0+b_1{x}_1+...+b_k{x}_k.$$

Again, the best fitting regression line is that which has the smallest sum of all of the error terms squared.

Although these equations have involved only a single error term in an equation, some statistical models involve multiple error terms. For instance, in structural equation modeling (SEM), multiple linear equations are used to estimate the model (which is why SEM also stands for simultaneous equation modeling) and each equation in the model has its own error term. In the case of multilevel modeling, different levels will have their own error terms.

The error term is sometimes confused with the concept of residuals. The error term refers to the difference in the actual outcome variables and the outcome variables predicted by the “true” model. However, when given a sample, one is only able to estimate the “true” statistical model (e.g., in a regression equation, one can only estimate the regression coefficients for the population based on the sample). Although the error term refers to the difference in actual outcomes and outcomes predicted by the “true” model, residuals refer to the difference in the actual outcomes and the outcomes predicted by the model estimated from a sample. Despite this subtle difference, one can conclude that a model estimated from a sample is the best fit for the sample if the estimation has small residuals. For example, in the case of a regression equation, the best fitting model has the estimates for b0, b1, through bk with the smallest sum of the squared residuals (squared residuals are used because the sum of the residuals would be 0).

...