Data in high dimensions: hypercube and hyperball

With many attributes, the data lives in a huge box, but the points fill only a tiny part of it. Distances and angles stop behaving the way they do in 2-D.

\(m\)

number of attributes = the dimension

\(L\)

half the side of the box: the biggest radius over all attributes

\(r\)

radius of the ball: the length of the longest point

ball vs sphere

the ball is the solid; the sphere is only its skin

The ball inside the cube shrinks

In 2-D a circle fills most of its square. As dimensions grow, the ball fills almost none of the cube, so almost all the room is in the corners.

Two dimensions:

\[ \frac{\operatorname{Vol}(\mathcal S_2(r))}{\operatorname{Vol}(\mathcal H_2(2r))} = \frac{\pi r^2}{4 r^2} = \frac{\pi}{4} = 78.5\% \]

Three dimensions:

\[ \frac{\operatorname{Vol}(\mathcal S_3(r))}{\operatorname{Vol}(\mathcal H_3(2r))} = \frac{4\pi r^3}{3 \times 8 r^3} = \frac{\pi}{6} = 52.4\% \]

The ratio keeps falling:

\[ \frac{\operatorname{Vol}(\mathcal S_m(r))}{\operatorname{Vol}(\mathcal H_m(2r))} \longrightarrow 0 \quad \text{as } m \longrightarrow \infty \]

• The volume of a high-dimensional hypercube is concentrated in its corners.
• The volume of the unit hypersphere decays to zero as \(m \to \infty\).
• All the volume of a hypersphere is concentrated in a thin shell as \(m \to \infty\).
• So the data tends to scatter on the corners of the cube and on the boundary of the sphere.

A random unit vector in \(\mathbb R^{30}\) almost never lands near one chosen direction, because the sphere's surface is spread over a huge number of “corners”. That is why 100,000 random tries still lose to the eigenvector, which the eigen-decomposition finds directly.

Linear PCA cannot see a curve

PCA only rotates the axes, so it cannot straighten a curved pattern. If you first transform the data with a non-linear function, PCA works again.

Before the transform the two eigenvalues are similar (3.41 and 1.73), so no single direction explains the data. After it, one eigenvalue dominates (7.04 vs 0.11): one direction is enough. The transform \(\phi\) is the first example of a feature map, the next card.

Feature map and the kernel trick

A feature map sends each point to a longer vector made of its squares and products. The kernel trick is that the dot product of two mapped vectors can be computed from the original dot product, without ever writing the long vectors.

\(\phi\)

the feature map: input vector in, longer feature vector out

\(\phi(\mathbf x)\)

the point in feature space (we usually never compute it)

\(K(\mathbf x, \mathbf y)\)

the kernel: a number, equal to \(\phi(\mathbf x)^T \phi(\mathbf y)\)

\(\sqrt 2\)

makes the cross term come out as \(2 x_1 x_2 y_1 y_2\) when squared

Kernel functions: polynomial and RBF

A kernel is any function of two points that equals a dot product in some feature space. The two you need are the polynomial kernel and the Gaussian (RBF) kernel.

\(p\)

degree of the polynomial (1 = linear, 2 = quadratic)

\(c\)

constant added before raising to the power; \(c = 0\) gives the homogeneous kernel \((\mathbf x^T \mathbf y)^p\)

\(\gamma\)

width parameter of the RBF kernel; bigger \(\gamma\) makes similarity fall faster with distance

\(\|\mathbf x - \mathbf y\|^2\)

squared Euclidean distance: sum of squared coordinate differences

The kernel matrix and its properties

The kernel matrix stores the kernel value for every pair of points. It is a table of similarities, and it is all that kernel methods ever look at.

Recipe: kernel matrix by hand

1. Write the points as columns of \(X\) and compute the Gram matrix \(X^T X\) (every pairwise dot product). For \(n\) points that is \(n(n+1)/2\) distinct dot products.
2. Polynomial kernel: apply \((c + \cdot)^p\) to every entry. The slide writes this with the entry-wise (Hadamard) product \(\circ\): \[ \mathbf K_2 = (1 + X^T X) \circ (1 + X^T X) \]
3. RBF kernel: first build the squared-distance matrix \(D = [\delta_{ij}]\) with \(\delta_{ij} = \|\mathbf x_i - \mathbf x_j\|^2\), then \[ \mathbf K_{RBF} = [e^{-\gamma \delta_{ij}}] \]
4. Check: symmetric, diagonal \(\ge 0\) (RBF diagonal is exactly 1).

\(\|\mathbf x_i - \mathbf x_j\|^2 = \mathbf x_i^T \mathbf x_i + \mathbf x_j^T \mathbf x_j - 2\, \mathbf x_i^T \mathbf x_j\). So the Gram matrix also gives you \(D\): diagonal entry plus diagonal entry minus twice the off-diagonal entry.

Norm, distance and cosine in feature space

The length of a mapped point, the distance between two mapped points, and the angle between them all come from three kernel values. You never need the mapped vectors themselves.

Each formula is the Lecture 1 formula with every dot product replaced by a kernel value: \(\mathbf a^T \mathbf b \to K(\mathbf a, \mathbf b)\). That substitution is the whole method.

Choosing γ and what RBF does to far-apart points

Customers compared with \(K(\mathbf x, \mathbf y) = \exp(-\gamma \|\mathbf x - \mathbf y\|^2)\). Two customers at input distance \(d = 2\) should have similarity 0.5. (a) Find \(\gamma\). (b) Show \(d_\phi^2 = 2 - 2K\). (c) Compute \(d_\phi\) for \(d = 1, 2, 4, 8\). (d) What happens to far-apart points?

1. \[ e^{-4\gamma} = \tfrac12 \]
2. \[ \gamma = \frac{\ln 2}{4} = 0.173 \] so \(K = 2^{-d^2/4}\).
3. \(K(\mathbf x, \mathbf x) = 1\) for every point, so \[ d_\phi^2 = 1 + 1 - 2K(\mathbf x, \mathbf y) = 2 - 2K \]

4. \(d\)	\(K = 2^{-d^2/4}\)	\(d_\phi = \sqrt{2 - 2K}\)
   1	0.841	0.564
   2	0.500	1.000
   4	0.0625	1.369
   8	≈ 0	1.414

5. Input distances 1 : 2 : 4 : 8 become 0.56 : 1 : 1.37 : 1.41. All far points look equally far, \(d_\phi \to \sqrt 2\). The RBF kernel is local; \(\gamma\) decides where “far” begins.

The mean in feature space and its length

The mean of the mapped points cannot be written down, but its length can: average all the entries of the kernel matrix. The distance from any point to that mean also comes from the kernel matrix.

\(\mathbf 1\)

a column of \(n\) ones; \(\mathbf 1^T \mathbf K \mathbf 1\) is the sum of all entries of \(\mathbf K\)

\(\mathbf 1_{n \times n}\)

the \(n \times n\) matrix of all ones (used in centring)

Centring in feature space: the centred kernel matrix

PCA needs the data centred at zero. You cannot subtract the mean from vectors you never computed, but you can fix the kernel matrix instead: subtract the row means and column means and add back the grand mean.

\(\hat{\mathbf K}\)

centred kernel matrix: dot products of the centred feature vectors

\(\mathbf H = \mathbf I - \frac1n \mathbf 1_{n \times n}\)

the centring matrix; \(\hat{\mathbf K} = \mathbf H \mathbf K \mathbf H\)

\(\bar r_i\)

mean of row \(i\) of \(\mathbf K\) (equal to the column mean, by symmetry)

\(\bar g\)

mean of all \(n^2\) entries of \(\mathbf K\), which is \(\|\boldsymbol\mu_\phi\|^2\)

Recipe: centre a kernel matrix by hand

1. Compute each row mean \(\bar r_i\).
2. Compute the grand mean \(\bar g\) (average of the row means).
3. For every entry: \[ \hat K_{ij} = K_{ij} - \bar r_i - \bar r_j + \bar g \]
4. Check: every row of \(\hat{\mathbf K}\) sums to 0, and \(\hat{\mathbf K}\) is symmetric. (Rows sum to 0 because the centred vectors add up to the zero vector.)

PCA recap, written with points as columns

This is the PCA you already know, but with the data stored as columns and the covariance written as a matrix product. Kernel PCA rewrites each of these lines using the kernel matrix.

Rows-as-samples there, columns-as-samples here. With columns, \(\frac1n X X^T\) is the covariance; with rows it is \(\frac1n X^T X\). Everything else is identical: see PCA as an optimisation and the PCA algorithm on the Lectures 3–4 page.

Kernel PCA: the idea and the obstacle

Do ordinary PCA, but on the mapped and centred points. The problem is that the covariance matrix in feature space is huge or infinite, so every step must be rewritten using the small n×n centred kernel matrix.

\(\boldsymbol\Sigma_\phi\)

covariance of the centred feature vectors (size: feature dimension squared)

\(\lambda\)

variance along a principal direction in feature space

\(\mathbf u\)

a principal direction in feature space (we will never write it explicitly)

Derivation: from Σ_φu = λu to K̂c = ηc

Each principal direction is a weighted sum of the data points, so finding it means finding n weights. Dotting the eigen-equation with each data point turns everything into kernel values, and the weights come out as an eigenvector of the centred kernel matrix.

PCA vs kernel PCA, and the linear kernel

Side by side, every PCA quantity has a kernel twin. With the linear kernel the twins are literally the same numbers, which is also a cheap way to do PCA when there are far more attributes than points.

Centring makes the \(n\) feature vectors add up to zero, so they span at most \(n - 1\) directions. One eigenvalue of \(\hat{\mathbf K}\) is always 0, with eigenvector \((1, 1, \dots, 1)^T\). With \(n = 2\) points there is exactly one component (HW2 Q3, Quiz 2); with \(n = 3\), two (L6 Example 3).

The kernel PCA algorithm

Build the kernel matrix, centre it, find its eigenvalues and unit eigenvectors, and scale each eigenvector by the square root of its eigenvalue to get the new coordinates.

Recipe for an exam question (n = 2 or 3 points)

1. Dot products of the points, then apply the kernel formula to each: \(\mathbf K\).
2. Row means and grand mean, then \(\hat K_{ij} = K_{ij} - \bar r_i - \bar r_j + \bar g\). Check rows sum to 0.
3. Eigenvalues of \(\hat{\mathbf K}\) from \(\det(\hat{\mathbf K} - \eta \mathbf I) = 0\). One of them is 0. Unit eigenvectors \(\mathbf v_i\).
4. Variances in feature space: \(\lambda_i = \eta_i / n\). Fraction kept by the first component: \(\eta_1 / \sum \eta_i\).
5. Coordinates of the training points on component \(i\): \(\mathbf a_i = \sqrt{\eta_i}\, \mathbf v_i\). Check: \(\frac1n \|\mathbf a_i\|^2 = \lambda_i\), and \(a_{1}^2 + a_{2}^2 + \dots = \hat K_{jj}\) for each point \(j\) when all components are kept.

Kernel PCA on Example 3, by hand

Start from the centred matrix of Example 3:

\[ \hat{\mathbf K} = \begin{bmatrix} 5 & -3 & -2 \\ -3 & 5 & -2 \\ -2 & -2 & 4 \end{bmatrix} \]

1. Eigenvectors from symmetry. Swapping \(\mathbf x_1 \leftrightarrow \mathbf x_2\) leaves \(\hat{\mathbf K}\) unchanged, so try \((1, -1, 0)^T\): \[ \hat{\mathbf K} \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix} = \begin{bmatrix} 8 \\ -8 \\ 0 \end{bmatrix} \] so \(\eta_1 = 8\).
2. \[ \hat{\mathbf K} \begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 \\ 6 \\ -12 \end{bmatrix} \] so \(\eta_2 = 6\).
3. \[ \hat{\mathbf K} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} \] so \(\eta_3 = 0\).
4. Variances: \[ \lambda_1 = \tfrac83, \quad \lambda_2 = 2 \]
5. Fraction of variance: \[ \tfrac{8}{14} = 57\%, \quad \tfrac{6}{14} = 43\% \] Two components capture 100%. With \(n = 3\) points at most \(n - 1 = 2\) can be nonzero.
6. Scaling \(\mathbf c_i = \mathbf v_i / \sqrt{\eta_i}\): \[ \mathbf c_1 = \frac{1}{\sqrt 2} \frac{1}{\sqrt 8} \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix} = \frac14 \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix} \]
7. \[ \mathbf c_2 = \frac{1}{\sqrt 6} \frac{1}{\sqrt 6} \begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix} = \frac16 \begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix} \]
8. Coordinates \(\mathbf a_i = \hat{\mathbf K} \mathbf c_i\): \[ \mathbf a_1 = \frac14 \begin{bmatrix} 8 \\ -8 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ -2 \\ 0 \end{bmatrix} \]
9. \[ \mathbf a_2 = \frac16 \begin{bmatrix} 6 \\ 6 \\ -12 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix} \]
10. Check the variances: \[ \tfrac13 (2^2 + 2^2 + 0^2) = \tfrac83 = \lambda_1 \]
11. \[ \tfrac13 (1^2 + 1^2 + 2^2) = 2 = \lambda_2 \]
12. PC 1 separates \(\mathbf x_1\) from \(\mathbf x_2\); PC 2 separates \(\mathbf x_3\) from the pair. In the input space \(\mathbf x_3\) looked like the outlier; kernel PCA spreads all three points apart.

Which formula, when

The formulas are given in the exam. The skill tested is picking the right one and showing the steps.

Rows of \(\hat{\mathbf K}\) sum to 0. One eigenvalue of \(\hat{\mathbf K}\) is 0. The kernel variances \(\eta_i / n\) add up to the mean of the diagonal of \(\hat{\mathbf K}\). For an RBF kernel every diagonal entry of \(\mathbf K\) is 1 and every entry lies in \((0, 1]\).