Correlation

If one wants to know the degree of a relationship, the correlation between two variables can be examined. Correlations can be quantified by computing a correlation coefficient. This entry first describes a concept central to correlation, covariance, and then discusses calculation and interpretation of correlation coefficients.

Covariance indicates the tendency in the linear relationship for two random variables to covary (or vary together) that is represented in deviations measured in the unstandardized units in which X and Y are measured. Specifically, it is defined as the expected product of the deviations of each of two random variables from its expected values or means.

The population covariance between two variables, X and Y, can be written by:

$$\begin{array}{c}cov(X,Y)=E[(X-E(X))(Y-E(Y))]\\ =E[XY-XE(Y)-E(X)Y-E(X)E(Y)]\\ =E(XY)-E(X)E(Y)-E(X)E(Y)-E(X)E(Y)\\ =E(XY)-E(X)E(Y)\end{array}$$

where E is the expected value or population mean.

Similarly, the sample covariance between x and y is given by:

$$\begin{array}{c}s(x,y)=\frac{1}{N-1}{\displaystyle \sum _{i=1}^n{(x)}_i}-\overline{x})(y_i-\overline{y})\\ =\frac{1}{N-1}\left[{\displaystyle \sum _{i=1}^n{x}_{}{}_i{y}_i-\frac{({\sum }^N{}_{i=1}{x}_i)({\sum }^N{}_{i=1}{y}_i)}{N}}\right]\end{array}$$

where N is the number of observations; $\overline{x},\overline{y}$ are the sample means of x and y.

When one interprets covariances, zero covariances indicate that variables are not linearly related. If they are nonlinearly associated or statistically independent, the covariance is zero. On the other hand, a nonzero covariance indicates the tendency of covarying. If the sign of covariance is positive, the two variables tend to vary in the same direction. If a covariance value is negative, the two variables tend to move in the opposite direction. The covariance is not independent of the unit used to measure x and y, and so magnitude of the covariance depends on the measurement units of two variables. Note that the nonzero covariance does not indicate causation and how strong the association is between two variables.

When considering a variance–covariance matrix, covariances or correlations among observed variables depend on the relationship between latent variables and linear composite variables (i.e., tests consisting of more than one item). The covariance between two composite variables is the sum of the elements of the covariance matrix. It can be written by:

$$\sigma (X,Y)=\sigma (X,Y)={\displaystyle \sum _{i=1}^p}{\displaystyle \sum _{j=1}^q\sigma (X_i,Y_j)}$$

where p, q are the numbers of variable in X and Y, respectively.

This most commonly computed correlation coefficient is a standardized index of linear association. From the covariance, the correlation coefficient for X and Y is calculated using the following equation:

$$r=\frac{{\displaystyle {\sum }_{i=1}^N\frac{x_i{y}_i}{N-1}}}{s(x,y)}=\frac{{\displaystyle {\sum }_{i=1}^N{x}_i{y}_i}}{(s_X{s}_Y)}$$

[Page 413]The product moment correlation coefficient was originally invented by Karl Pearson in 1895 based on the studies conducted by Francis Galton and J. D. Hamilton Dickson in the 1880s. The correlation coefficient ranges from −1 to 1. If its value is 0, the variables have no linear relationship; and if its value is −1 or 1, each variable is perfectly predicted by the other. Its sign indicates the direction of the relationship.