Canonical Correlation

Affine Invariance

A simple observation suggests that if we scale either u or v or both, the objective does not change. That is, if u 7→ αu and/or v 7→ βv, the objective does not change. This has a profound implication—canonical correlation is statistical measure that is invariant to affine transformations, unlike other measures, for instance, those optimized by principal component analysis and partial least squares.

CCA as a constrained optimization problem: Because the CCA objective is affine invariant, we might as well choose the scaling coefficients, α, β ∈R such that:

$$\mathbb{E}\left[u^{\top }x{x}^{\top }u\right]=1,\mathbb{E}\left[v^{\top }y{y}^{\top }v\right]=1.$$

This yields an equivalent constrained optimization problem:

$$\begin{array}{ll}\text{maximize}\hfill & \mathbb{E}\left(x,y\right)\left[u^{\tau }x{y}^{\tau }v\right],\hfill \\ \hfill & u\in {\text{R}}^{dx},v\in {\text{R}}^{dy}\hfill \\ \text{subject}\text{to}\hfill & {\mathbb{E}}_x\left[u^{\tau }x{x}^{\tau }u\right]=1,{\mathbb{E}}_y\left[v^{\tau }y{y}^{\tau }v\right]=1.\hfill \end{array}$$

CCA Solution

We can show, using the Lagrange multiplier’s method, that the solution to the constrained optimization problem above is given by choosing u to [Page 235]be the top eigenvector of C−xx1 Cxy C−yy1 Cyx and choosing v = C−yy√1Cλyxu.

Simultaneous Solution to CCA

Subsequent CCA directions can be found by deflation by constraining them to be uncorrelated to previous ones. Alternatively, we can solve for to pk CCA dimensions simultaneously by solving for the following optimization problem:

Find $U\in {ℝ}^{d_x\times k},V\in {ℝ}^{d_y\times k}$ that

$$\begin{array}{c}\text{maximize}:{\mathbb{E}}_{\left(x,y\right)\sim D}\left[\text{trace}\left(U^{\top }x{y}^{\top }V\right)\right]\\ \text{subject}\text{to}:\mathbb{E}\left[U^{\top }x{x}^{\top }U\right]=I_k,\mathbb{E}\left[V^{\top }y{y}^{\top }V\right]=I_k.\end{array}$$

In other words, CCA can be posed as the problem of finding the most correlated k-dimensional subspace of D and columns of U are given as the top-k eigenvectors of C−xx1 Cxy C−yy1 Cyx 29th Conference on Neural Information Processing Systems (2016), Barcelona, Spain.

CCA as Minimizing Reconstruction Error

Given n data (xi,yi) in Rdx × Rdy, from an unknown D, find U ∈ Rdx × k, V ∈ Rdy ×k

$$\begin{array}{c}\text{that}\text{minimize}:{\mathbb{E}}_{x,y\sim D}\parallel U^{\top }x-V^{\top }y{\parallel }_2^2,\\ \text{subject}\text{to}:U>C_{xx}U=I_k,V>C_{yy}V=I_k.\end{array}$$

Expand the objective:

$$\begin{array}{c}\mathbb{E}\parallel U^{T}x-V^{T}y{\parallel }_2^2\\ =\mathbb{E}\left[x^{\top }U{U}^{\top }x\right]-2\mathbb{E}\left[x^{\top }U{V}^{\top }y\right]+\mathbb{E}\left[y^{\top }V{V}^{\top }y\right]\\ =E^h\text{trace}{U}^{\top }x{x}^{\top }U)]\\ -2\mathbb{E}\left[\text{trace}{U}^{\top }x{y}^{\top }V\right]+E^h\text{trace}{V}^{\top }y{y}^{\top }V)].\end{array}$$

History

CCA is a classical technique in multivariate statistics, proposed by Harold Hotelling in 1935, for measuring correlations between two random vectors. Hotelling first studied this problem to predict success in college. In his 1935 article, titled The Most Predictable Criterion,” he argued that no single regression equation provides fully adequate solution and that regressing on the dependent variate with the largest multiple correlation with predictors yields best accuracy.

Applications

CCA is widely used in multivariate analysis, finance, management sciences, chemometrics, bioinformatics, and neuroscience. CCA has been successfully applied to various tasks in speech, natural language processing, and computer vision. CCA admits a nonstandard stochastic optimization problem where not only the objective but the constraints are stochastic or equivalently the objective is a ratio of two expectations rather than an expectation of a loss function. Consequently, the CCA objective does not decompose over the sample and designing stochastic approximation algorithms for CCA remains a challenging open problem.

...