Convergence

Convergence is a process in statistical analysis describing a series of calculations or guesses for the purpose of ultimately producing a very precise estimate. A simple example of several related equations illustrates the process. Imagine being tasked with solving the following set of equations for X and Y:

$$X+Y=5,$$

$$X-Y=3,$$

$$2X+3Y=12.$$

Given only Equations 1 and 2, the task easily yields X = 4 and Y = 1. However, these values do not satisfy Equation 3, for which substituting X = 4 and Y = 1 would yield 11 rather than 12. In fact, there is no exact solution for X and Y that satisfies all three equations. Still, one may wish to know what estimates of X and Y, $\tilde{X}$ and Ỹ, respectively, make all three equations as close to true as possible.

What “close” means in one equation is clear; for Equation 1, for example, $\tilde{X}+\tilde{Y}$ should yield a value close to 5. For more general purposes, however, close must be operationalized across the set of equations by specifying a function that yields a single numerical value operationalizing the discrepancy between the equations’ outcome values (5, 3, 12) and the outcome values expected [Page 404]based on the estimates $\tilde{X}$ and Ỹ. Such a discrepancy function, or fit function, may then be used to guide the derivation of optimal values for those estimates.

A simple discrepancy function example, representing an unweighted least squares criterion, would be:

$$F={[5-(\tilde{X}+\tilde{Y})]}^2+{[3-(\tilde{X}-\tilde{Y})]}^2+{[12-(2\tilde{X}+3\tilde{Y})]}^2.$$

Using this function, we seek $\tilde{X}$ and Ỹ values minimizing F. Readers familiar with multivariable calculus recognize that this could be accomplished analytically, setting to zero the partial derivatives of F with respect to $\tilde{X}$ and Ỹ and solving. Computers, however, are less adept at analytical solutions; fortunately, they are good at using algorithms that employ iterative strategies to derive estimates for unknown quantities.

After choosing initial start values for $\tilde{X}$ and Ỹ, a computer changes those estimates incrementally, moving in those directions that make F smaller. The process continues adaptively through several iterations, altering the estimates in typically smaller increments until F reaches convergence. That is, the algorithm stops when F can no longer meaningfully decrease given its incremental changes in $\tilde{X}$ and Ỹ, ideally reaching a value close to the analytical minimum; the resulting empirical values of $\tilde{X}$ and Ỹ constitute the estimates according to the criterion used to define F.

Although this is just a simple example, it represents a process that occurs throughout much of statistics. Maximum likelihood estimation, for example, employed across many applications (e.g., logistic regression, item response theory, structural equation modeling), seeks estimates for model parameters that optimize a discrepancy function characterizing the likelihood of all observations within a sample of data.

Statistical methods employ many different types of discrepancy functions as well as different search algorithms to optimize them. An algorithm might also fail to converge upon a solution, a result more common with complex models and models with very poor fit to the data. Alternatively, an algorithm might converge but reach a local minimum in the discrepancy function rather than the global minimum. This underscores the importance of choosing multiple sets of start values to ensure convergence occurs and that it is to a globally optimum solution for the parameter estimates of interest to the researcher.

...