1. \[ \mathbf u^T\mathbf v = 2 + 4 + 6 = 12 \]
2. \[ \|\mathbf u\| = \sqrt{14} \approx 3.742, \qquad \|\mathbf v\| = \sqrt{12} \approx 3.464 \]
3. \[ \cos\theta = \frac{12}{\sqrt{14}\sqrt{12}} = \frac{12}{\sqrt{168}} \approx 0.926 \]
4. \[ \theta = \cos^{-1}(0.926) \approx 22.2^\circ \]

1. \[ \mathrm{proj}_{\mathbf v}(\mathbf u) = \frac{\mathbf u^T\mathbf v}{\mathbf v^T\mathbf v}\,\mathbf v = \frac{12}{12}\,\mathbf v = \begin{bmatrix} 2 \\ 2 \\ 2 \end{bmatrix} \]
2. \[ \mathbf u_\perp = \mathbf u - \mathrm{proj}_{\mathbf v}(\mathbf u) = \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} \]
3. \[ \mathbf u_\perp^T\mathbf v = -2 + 0 + 2 = 0 \quad\checkmark \]
4. \[ (2,2,2) + (-1,0,1) = (1,2,3) = \mathbf u \quad\checkmark \]

Subtracting the projection is exactly how HW1 Q7 forced the second random direction to be orthogonal to the first, and how Lecture 7 splits \(\mathbf X\) into \(\mu_X\mathbf 1\) plus a centred part.

1. \[ \|\mathbf u\|_1 = 1 + 2 + 3 = 6, \qquad \|\mathbf u\|_\infty = \max(1, 2, 3) = 3 \]
2. (i) \[ 2 + 2 - 4 = 0 \quad\text{orthogonal} \]
3. (ii) \[ 1 + 2 - 3 = 0 \quad\text{orthogonal} \]
4. (iii) \[ 3 + 1 - 2 = 2 \ne 0 \quad\text{not orthogonal} \]

Orthogonal means dot product zero, nothing else. Equal lengths or "nearly 90°" (HW1 Q1 had 86.9°) do not count.

1. \[ \int_0^1 c(1 - y)\,dy = c\Big[y - \tfrac{y^2}{2}\Big]_0^1 = \tfrac{c}{2} = 1 \;\Rightarrow\; c = 2 \]
2. \[ F(y) = \int_0^y 2(1 - t)\,dt = 2y - y^2, \qquad 0 \le y \le 1 \]
3. \[ P(Y \le 0.5) = F(0.5) = 1 - 0.25 = 0.75 \]

1. Median: \(F(\nu) = \tfrac12\): \[ \nu^2 - 2\nu + \tfrac12 = 0 \;\Rightarrow\; \nu = 1 - \sqrt{\tfrac12} \approx 0.293 \]
2. \[ E[Y] = \int_0^1 2y(1 - y)\,dy = 2\Big(\tfrac12 - \tfrac13\Big) = \tfrac13 \approx 0.333 \]
3. The density is tallest at 0 and tapers to the right, so the long right tail pulls the mean above the median.

Take the root between 0 and 1. The other root, \(1 + \sqrt{1/2}\), is outside the range.

1. \[ F(0) = 0.2, \quad F(1) = 0.7, \quad F(2) = 1 \]
2. \[ P(X \ge 1) = 1 - F(0) = 0.8 \]
3. \[ E[X] = 0(0.2) + 1(0.5) + 2(0.3) = 1.1 \]
4. \[ E[X^2] = 0 + 0.5 + 4(0.3) = 1.7 \]
5. \[ \mathrm{Var}(X) = 1.7 - 1.1^2 = 0.49 \]

\(P(X \ge 1)\) is \(1 - F(0)\), not \(1 - F(1)\). For a discrete variable the strict and non-strict inequalities differ by one whole jump; the formula sheet lists all four cases.

1. Sort: \[ (1, 1, 2, 3, 4, 5, 6, 9) \]
2. \[ \bar x = \tfrac{31}{8} = 3.875 \]
3. \(n = 8\) even: \[ \hat\nu = \tfrac12(x_4 + x_5) = \tfrac12(3 + 4) = 3.5 \]
4. Lower half \((1,1,2,3)\), upper half \((4,5,6,9)\): \[ q_1 = \tfrac{1 + 2}{2} = 1.5, \qquad q_3 = \tfrac{5 + 6}{2} = 5.5 \]
5. \[ \mathrm{IQR} = 5.5 - 1.5 = 4, \qquad R = 9 - 1 = 8 \]
6. \[ \hat\sigma^2 = \tfrac18\sum(x_i - 3.875)^2 = \tfrac{52.875}{8} \approx 6.61 \]

With \(n - 1\) the variance is \(52.875/7 \approx 7.55\). The exam will say which denominator to use; if it does not, state your choice.

1. \[ \bar x = \tfrac{112}{8} = 14, \qquad \hat\nu = 3.5 \text{ (unchanged)} \]
2. One extreme value moved the mean past every other observation; the median did not move at all. That is what "the median is robust to outliers" means.

1. \[ \hat{\boldsymbol\mu} = (5, 3) \]
2. Deviations: \[ X - 5 = (-3, -1, 1, 3), \qquad Y - 3 = (0, -2, 0, 2) \]
3. \[ \hat\sigma_{XX} = \tfrac14(9 + 1 + 1 + 9) = 5 \]
4. \[ \hat\sigma_{YY} = \tfrac14(0 + 4 + 0 + 4) = 2 \]
5. \[ \hat\sigma_{XY} = \tfrac14(0 + 2 + 0 + 6) = 2 \]
6. \[ \hat\Sigma = \begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix} \]

1. \[ r = \frac{2}{\sqrt{5 \cdot 2}} = \frac{2}{\sqrt{10}} \approx 0.632 \]

1. \[ \operatorname{tr}\hat\Sigma = 7, \qquad \det\hat\Sigma = 10 - 4 = 6 \]
2. \[ \lambda^2 - 7\lambda + 6 = 0 \;\Rightarrow\; (\lambda - 6)(\lambda - 1) = 0 \]
3. \[ \lambda_1 = 6, \qquad \lambda_2 = 1 \]
4. \((\hat\Sigma - 6I)\mathbf v = 0\): first row \(-v_1 + 2v_2 = 0\), so \(v_1 = 2v_2\): \[ \mathbf v_1 = \tfrac{1}{\sqrt5}\begin{bmatrix} 2 \\ 1 \end{bmatrix} \approx \begin{bmatrix} 0.894 \\ 0.447 \end{bmatrix} \]
5. \[ \frac{6}{7} \approx 85.7\% \]

\(\lambda_1\lambda_2 = 6 = \det\), \(\lambda_1 + \lambda_2 = 7 = \operatorname{tr}\). The second eigenvector is perpendicular: \(\tfrac{1}{\sqrt5}(-1, 2)\).

1. \[ \sigma = \sqrt5 \approx 2.236 \]
2. \[ z = \frac{9 - 5}{2.236} = 1.79 \;\Rightarrow\; P(X \gt 9) = 1 - 0.9633 = 0.0367 \]
3. \[ z_{\pm} = \frac{7 - 5}{2.236} = 0.89, \qquad \frac{3 - 5}{2.236} = -0.89 \]
4. \[ P(3 \le X \le 7) = \Phi(0.89) - \Phi(-0.89) = 2(0.8133) - 1 = 0.6266 \]

\(N(5, 5)\) means variance 5, so \(\sigma = \sqrt5\). Using 5 as the standard deviation is a one-mark loss that cascades through the whole part.

1. \[ \boldsymbol\mu = \tfrac14(12, 12) = (3, 3) \ne (0, 0) \]
2. Not centred. Subtract the mean: \[ \mathbf z_1 = (-2,-2), \quad \mathbf z_2 = (-1, 0), \quad \mathbf z_3 = (0, -1), \quad \mathbf z_4 = (3, 3) \]

The Quiz 2 key gave one full point for "checks/uses the mean". The data there happened to be centred already; here it is not, and every later part depends on this step.

1. \[ a_i = \frac{z_{i1} + z_{i2}}{\sqrt2} \]
2. \[ a_1 = \tfrac{-4}{\sqrt2} = -2.828, \quad a_2 = \tfrac{-1}{\sqrt2} = -0.707, \quad a_3 = -0.707, \quad a_4 = \tfrac{6}{\sqrt2} = 4.243 \]
3. Check: \(\sum a_i = 0\), as it must for centred data.

1. The eigenvalue equals the variance of the projected data (mean 0): \[ \lambda_1 = \tfrac14\sum a_i^2 = \tfrac14\big(8 + 0.5 + 0.5 + 18\big) = \tfrac{27}{4} = 6.75 \]
2. \[ \Sigma = \tfrac14\begin{bmatrix} 4 + 1 + 0 + 9 & 4 + 0 + 0 + 9 \\ 13 & 4 + 0 + 1 + 9 \end{bmatrix} = \begin{bmatrix} 3.5 & 3.25 \\ 3.25 & 3.5 \end{bmatrix} \]
3. \[ \lambda_2 = \operatorname{tr}\Sigma - \lambda_1 = 7 - 6.75 = 0.25 \]
4. Check with the shortcut for equal diagonals: \(3.5 \pm 3.25 = 6.75,\ 0.25\) ✓. Fraction on PC 1: \(6.75 / 7 = 96.4\%\).

The quiz key's intended method was the variance of the coordinates, and it gave full credit for the \(\Sigma\) route too. Showing both takes thirty seconds and makes the answer self-checking.

1. \[ \tilde{\mathbf x}_i = \boldsymbol\mu + a_i\mathbf u_1 = (3,3) + \tfrac{z_{i1} + z_{i2}}{2}(1,1) \]
2. \[ \tilde{\mathbf x}: \quad (1,1),\ (2.5,2.5),\ (2.5,2.5),\ (6,6) \]
3. Errors: \[ (0,0),\ (-0.5, 0.5),\ (0.5,-0.5),\ (0,0) \]
4. \[ \mathrm{MSE} = \tfrac14(0 + 0.5 + 0.5 + 0) = 0.25 = \lambda_2 \]

1. \[ \mathbf u_1^T\mathbf x_i = a_i + \mathbf u_1^T\boldsymbol\mu = a_i + \tfrac{6}{\sqrt2} = a_i + 4.243 \]
2. \[ (1.414,\ 3.536,\ 3.536,\ 8.485) \]
3. The variance is unchanged (adding a constant does not change spread), but the coordinates are wrong, the reconstruction \(\boldsymbol\mu + a_i\mathbf u_1\) is wrong, and the marks for (b) and (d) are lost.

This is exactly the error in the posted 2025 HW2 Q1 key, flagged in amber on the PCA study page.

1. Products \(x_ix_j\): \[ \begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 2 \\ 0 & 2 & 4 \end{bmatrix} \]
2. Add 1 and square each entry: \[ \mathbf K = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 4 & 9 \\ 1 & 9 & 25 \end{bmatrix} \]

1. \[ (1 + xy)^2 = 1 + 2xy + x^2y^2 \;\Rightarrow\; \phi(x) = (1,\ \sqrt2\,x,\ x^2)^T \]
2. \[ \phi(0) = (1, 0, 0), \quad \phi(1) = (1, \sqrt2, 1), \quad \phi(2) = (1, 2\sqrt2, 4) \]
3. \[ \phi(1)^T\phi(2) = 1 + 4 + 4 = 9 = K_{23} \quad\checkmark \]
4. \[ \phi(2)^T\phi(2) = 1 + 8 + 16 = 25 = K_{33} \quad\checkmark \]

1. \[ \|\boldsymbol\mu_\phi\|^2 = \frac{1}{9}\,\mathbf 1^T\mathbf K\mathbf 1 = \frac{1 + 1 + 1 + 1 + 4 + 9 + 1 + 9 + 25}{9} = \frac{52}{9} \approx 5.78 \]
2. \[ \|\boldsymbol\mu_\phi\| \approx 2.40 \]

1. \[ \|\phi(x_3) - \boldsymbol\mu_\phi\|^2 = K_{33} - \frac{2}{n}\sum_j K_{3j} + \|\boldsymbol\mu_\phi\|^2 \]
2. \[ \sum_j K_{3j} = 1 + 9 + 25 = 35 \]
3. \[ = 25 - \tfrac{2}{3}(35) + \tfrac{52}{9} = \tfrac{225 - 210 + 52}{9} = \tfrac{67}{9} \approx 7.44 \]
4. \[ \|\phi(x_3) - \boldsymbol\mu_\phi\| \approx 2.73 \]

\(67/9\) is \(\hat K_{33}\), the diagonal entry of the centred kernel matrix: the squared distance of a point from the mean is its centred self-similarity.

1. \[ \cos\theta_\phi = \frac{K_{23}}{\sqrt{K_{22}K_{33}}} = \frac{9}{\sqrt{100}} = 0.9 \]
2. \[ \|\phi(x_2) - \phi(x_3)\| = \sqrt{4 + 25 - 18} = \sqrt{11} \approx 3.32 \]

1. \[ e^{-4\gamma} = \tfrac12 \;\Rightarrow\; \gamma = \tfrac{\ln 2}{4} \approx 0.173 \]
2. \(K(x,x) = 1\) for every point, so \[ d_\phi^2 = 2 - 2K(x,y) = 2 - 1 = 1, \qquad d_\phi = 1 \]
3. As the input distance grows, \(K \to 0\) and \(d_\phi \to \sqrt2\): all far-away points look equally far. The RBF kernel is local, and \(\gamma\) decides where "far" begins.

1. \[ \mathbf K = \begin{bmatrix} 2 & 2 \\ 2 & 4 \end{bmatrix} \]

1. Row means \(\bar r = (2, 3)\), grand mean \(\bar g = \tfrac{10}{4} = 2.5\).
2. \[ \hat K_{11} = 2 - 2 - 2 + 2.5 = 0.5 \]
3. \[ \hat K_{12} = 2 - 2 - 3 + 2.5 = -0.5 \]
4. \[ \hat K_{22} = 4 - 3 - 3 + 2.5 = 0.5 \]
5. \[ \hat{\mathbf K} = \begin{bmatrix} 0.5 & -0.5 \\ -0.5 & 0.5 \end{bmatrix} \]

Or \(\hat{\mathbf K} = \mathbf H\mathbf K\mathbf H\) with \(\mathbf H = \tfrac12\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix}\). Same matrix.

1. \[ (0.5 - \eta)^2 - 0.25 = \eta(\eta - 1) = 0 \]
2. \[ \eta_1 = 1,\ \mathbf v_1 = \tfrac{1}{\sqrt2}(1, -1)^T; \qquad \eta_2 = 0,\ \mathbf v_2 = \tfrac{1}{\sqrt2}(1, 1)^T \]
3. \[ \lambda_1 = \tfrac{\eta_1}{n} = 0.5, \qquad \lambda_2 = 0 \]

1. \[ \mathbf c_1 = \tfrac{1}{\sqrt1}\cdot\tfrac{1}{\sqrt2}(1,-1)^T = (0.707, -0.707)^T \]
2. \[ \mathbf a = \hat{\mathbf K}\mathbf c_1 = \sqrt{\eta_1}\,\mathbf v_1 = (0.707, -0.707) \]
3. Check: \(\tfrac12(0.5 + 0.5) = 0.5 = \lambda_1\) ✓

1. \[ \boldsymbol\mu = (1.5, 0.5), \qquad \mathbf z_1 = (-0.5, 0.5), \quad \mathbf z_2 = (0.5, -0.5) \]
2. \[ \Sigma = \tfrac12\begin{bmatrix} 0.5 & -0.5 \\ -0.5 & 0.5 \end{bmatrix} = \begin{bmatrix} 0.25 & -0.25 \\ -0.25 & 0.25 \end{bmatrix} \]
3. \[ \lambda = 0.25 \pm 0.25 = 0.5,\ 0 \quad\checkmark \]
4. \(\mathbf u_1 = \tfrac{1}{\sqrt2}(1, -1)\), coordinates \(\mathbf u_1^T\mathbf z_i = (-0.707, 0.707)\): the same as (d) up to sign ✓

\(\mathbf X^T\mathbf X\) (which is \(\hat{\mathbf K}\) for centred data) and \(\mathbf X\mathbf X^T = n\Sigma\) share their nonzero eigenvalues, so \(\eta = n\lambda\). Kernel PCA with the linear kernel is PCA; nonlinear kernels are where it adds something.

1. \[ \mu_1 = 3, \qquad \mu_Y = 10 \]
2. Centred: \(x_1 - 3 = (-2,-1,0,1,2)\), \(y - 10 = (-3,-2,2,2,1)\).
3. \[ w = \frac{6 + 2 + 0 + 2 + 2}{4 + 1 + 0 + 1 + 4} = \frac{12}{10} = 1.2, \qquad b = 10 - 1.2(3) = 6.4 \]
4. \[ \hat{\mathbf y} = (7.6, 8.8, 10, 11.2, 12.4), \qquad \boldsymbol\epsilon = (-0.6, -0.8, 2, 0.8, -1.4) \]
5. \[ \mathrm{SSE} = 0.36 + 0.64 + 4 + 0.64 + 1.96 = 7.6 \]

1. \(\mu_2 = 2\), centred \(x_2 - 2 = (0, -1, 0, 1, 0)\).
2. \[ \bar{\mathbf D}^T\bar{\mathbf D} = \begin{bmatrix} 10 & 2 \\ 2 & 2 \end{bmatrix} \quad (\text{cross term: } 0 + 1 + 0 + 1 + 0 = 2) \]
3. \[ \bar{\mathbf D}^T\bar{\mathbf Y} = \begin{bmatrix} 12 \\ 0 + 2 + 0 + 2 + 0 \end{bmatrix} = \begin{bmatrix} 12 \\ 4 \end{bmatrix} \]
4. \[ \det = 20 - 4 = 16, \qquad (\bar{\mathbf D}^T\bar{\mathbf D})^{-1} = \tfrac{1}{16}\begin{bmatrix} 2 & -2 \\ -2 & 10 \end{bmatrix} \]
5. \[ \mathbf w = \tfrac{1}{16}\begin{bmatrix} 24 - 8 \\ -24 + 40 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} \]
6. \[ b = \mu_Y - w_1\mu_1 - w_2\mu_2 = 10 - 3 - 2 = 5 \]
7. \[ \hat y = 5 + x_1 + x_2 \]

Matrix multiplication was the TA's third listed problem after Quiz 2. Write the 2 × 2 inverse formula from the sheet, swap the diagonal, negate the off-diagonal, divide by the determinant, and multiply row by column slowly.

1. \[ \hat{\mathbf y} = (8, 8, 10, 12, 12), \qquad \boldsymbol\epsilon = (-1, 0, 2, 0, -1), \qquad \mathrm{SSE} = 6 \lt 7.6 \]
2. \(x_1\) and \(x_2\) are correlated (\(r = 0.447\)). With \(x_2\) absent, the single-predictor fit gave \(x_1\) credit for some of the effect of \(x_2\). A coefficient's meaning depends on which other predictors are in the model, and adding a predictor can never increase the training SSE.