Dimension Reduction

Unsupervised Dimension Reduction

Unsupervised dimension reduction treats all variables equally without specifying a response. The analysis usually has a natural definition about the information of interest. Unsupervised dimension reduction methods find a new set of a smaller number of variables that either provides a simpler presentation or discovers intrinsic structure in the data while retaining most of the important information. Listed below are only a few of the most widely used techniques.

Principal component analysis (PCA) finds a few orthogonal linear combinations of the original variables with the largest variances; these linear combinations are the principal components that would be retained for subsequent analyses. In PCA, the information is the variation within the data. Usually, principal components are sorted in descending order according to their variations. The number of principal components that should be included in the analysis depends on how much variation should be preserved.

Factor analysis assumes that a set of variables establishes the relationships among themselves through a smaller set of common factors. It estimates the common factors with assumptions about the variance-covariance structure.

Canonical correlation analysis identifies and measures the association between two sets of random variables. Often, it finds one linear combination of variables for each set, where these two new variables have the largest correlation.

Correspondence analysis is a graphical tool for an exploratory data analysis of a contingency table. It projects the rows and columns as points into a plot, where rows (columns) have a similar profile if their corresponding points are close together.

Projection pursuit defines a projection index that measures the “interestingness” of a direction. Then, it searches for the direction maximizing the index.

Multidimensional scaling finds a projection of the data into a smaller dimensional space so that the distances among the points in the new space reflect the proximities in the original data.

Supervised Dimension Reduction

Supervised dimension reduction techniques generally are applied in regression. A response Y is specified that can be one random variable, one random vector, or even a curve. The predictor vector X is p-dimensional. The object of interest is the relation between the response and the predictors, which is often summarized as Y = f(X, ∊), where ∊ denotes the error term. Some specific structures are imposed to facilitate the estimation of the function. Dimension reduction is a crucial part of the modeling process. For example, ordinary least squares regression can be considered as a special case of dimension reduction in regression.

...