1. \[ \|\mathbf q\| = \sqrt2, \quad \|\mathbf d_1\| = \sqrt5, \quad \|\mathbf d_2\| = \sqrt6, \quad \|\mathbf d_3\| = 3 \]
2. \[ \cos(\mathbf q, \mathbf d_1) = \frac{3}{\sqrt2\sqrt5} = \frac{3}{\sqrt{10}} \approx 0.949 \]
3. \[ \cos(\mathbf q, \mathbf d_2) = \frac{2}{\sqrt2\sqrt6} = \frac{2}{\sqrt{12}} \approx 0.577 \]
4. \[ \cos(\mathbf q, \mathbf d_3) = \frac{0}{3\sqrt2} = 0 \]
5. Ranking: \(\mathbf d_1 \gt \mathbf d_2 \gt \mathbf d_3\). \(\mathbf d_3\) shares no words with the query, so it is orthogonal to it.

1. \[ \|\mathbf q - \mathbf d_1\| = \|(-1, 0, 0)\| = 1 \]
2. \[ \|\mathbf q - \mathbf d_2\| = \|(0, 0, -2)\| = 2 \]
3. \[ \|\mathbf q - \mathbf d_3\| = \|(1, 1, -3)\| = \sqrt{11} \approx 3.32 \]
4. Same ranking: \(\mathbf d_1\), then \(\mathbf d_2\), then \(\mathbf d_3\).

1. \[ \mathrm{proj}_{\mathbf g}(\mathbf d_2) = \frac{\mathbf d_2^T\mathbf g}{\mathbf g^T\mathbf g}\,\mathbf g = \frac{2}{2}\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} \]
2. \[ \mathbf d_2 - \mathrm{proj}_{\mathbf g}(\mathbf d_2) = \begin{bmatrix} 0 \\ 0 \\ 2 \end{bmatrix} \]
3. What survives is orthogonal to \(\mathbf g\): only the "recipe" count remains.

Cosine ignores length. A long document that repeats the query's words many times has a large Euclidean distance from a short query simply because it is long, yet it points in the same direction. Cosine compares direction only, so document length does not penalise relevance. (Scaling \(\mathbf d_1\) to \((4, 2, 0)\) leaves its cosine at 0.949 but raises its distance from 1 to \(\sqrt{10}\).)

1. \[ \bar x = \tfrac{40}{8} = 5 \]
2. Already sorted; \(n = 8\): \[ \hat\nu = \tfrac12(4 + 5) = 4.5 \]
3. Halves \((2,4,4,4)\) and \((5,5,7,9)\): \[ q_1 = \tfrac12(4 + 4) = 4, \qquad q_3 = \tfrac12(5 + 7) = 6 \]
4. \[ \mathrm{IQR} = 6 - 4 = 2 \]

1. Deviations \((-3, -1, -1, -1, 0, 0, 2, 4)\): \[ \sum(x_i - 5)^2 = 9 + 1 + 1 + 1 + 0 + 0 + 4 + 16 = 32 \]
2. \[ \hat\sigma^2 = \tfrac{32}{8} = 4, \qquad \hat\sigma = 2 \]
3. \[ s^2 = \tfrac{32}{7} \approx 4.571 \]

\(\hat\sigma^2\) is the squared length of the centred vector divided by \(n\) (Lecture 2 slide 23). The \(n - 1\) version corrects the small downward bias of the \(\tfrac1n\) estimate.

1. \[ \hat F(4) = \frac{\#\{x_i \le 4\}}{8} = \frac{4}{8} = 0.5 \]
2. \[ \hat F(6) = \frac{6}{8} = 0.75 \]

1. \[ z = \frac{9 - 5}{2} = 2 \;\Rightarrow\; P(X \ge 9) = 1 - \Phi(2) = 0.0228 \]
2. \[ P(3 \lt X \le 7) = \Phi(1) - \Phi(-1) = 2(0.8413) - 1 = 0.6826 \]

The second answer is the "68% within one standard deviation" rule from Lecture 3 slide 14. Knowing it lets you check the table lookup.

1. \[ c(1 + 4 + 9) = 1 \;\Rightarrow\; c = \tfrac{1}{14} \]
2. \[ F(2) = \tfrac{1 + 4}{14} = \tfrac{5}{14} \approx 0.357 \]
3. \[ E[X] = \tfrac{1 + 8 + 27}{14} = \tfrac{36}{14} \approx 2.571 \]

1. \[ F(x) = x^3 \]
2. \[ P(X \gt 0.5) = 1 - 0.125 = 0.875 \]
3. \[ \nu^3 = \tfrac12 \;\Rightarrow\; \nu = 0.5^{1/3} \approx 0.794 \]

1. \[ \hat{\boldsymbol\mu} = \tfrac14(8, 8, 4) = (2, 2, 1) \]

1. \[ \mathbf Z = \begin{bmatrix} -1 & 0 & -1 \\ 0 & -2 & 0 \\ 1 & 0 & 1 \\ 0 & 2 & 0 \end{bmatrix} \]

Column 1 and column 3 of \(\mathbf Z\) are identical. Notice it now; it explains everything that follows.

1. Column dot products: \(\mathbf z_1\!\cdot\!\mathbf z_1 = 2\), \(\mathbf z_2\!\cdot\!\mathbf z_2 = 8\), \(\mathbf z_3\!\cdot\!\mathbf z_3 = 2\), \(\mathbf z_1\!\cdot\!\mathbf z_3 = 2\), \(\mathbf z_1\!\cdot\!\mathbf z_2 = 0\), \(\mathbf z_2\!\cdot\!\mathbf z_3 = 0\).
2. \[ \hat\Sigma = \tfrac14\begin{bmatrix} 2 & 0 & 2 \\ 0 & 8 & 0 \\ 2 & 0 & 2 \end{bmatrix} = \begin{bmatrix} 0.5 & 0 & 0.5 \\ 0 & 2 & 0 \\ 0.5 & 0 & 0.5 \end{bmatrix} \]

Marks: 1 for the diagonal, 1 for the off-diagonals in the right places, 1 for the \(\tfrac1n\) and symmetry.

1. \[ \text{Total variance} = \operatorname{tr}\hat\Sigma = 0.5 + 2 + 0.5 = 3 \]
2. Attribute 2 is uncoupled, so \(\mathbf e_2 = (0, 1, 0)\) is an eigenvector: \[ \lambda = 2 \]
3. The remaining \(2 \times 2\) block \(\begin{bmatrix} 0.5 & 0.5 \\ 0.5 & 0.5 \end{bmatrix}\) has eigenvalues \(0.5 \pm 0.5\): \[ \lambda = 1 \text{ with } \tfrac{1}{\sqrt2}(1, 0, 1), \qquad \lambda = 0 \text{ with } \tfrac{1}{\sqrt2}(1, 0, -1) \]
4. Ordered: \[ \lambda_1 = 2, \quad \lambda_2 = 1, \quad \lambda_3 = 0; \qquad \sum\lambda_i = 3 = \operatorname{tr}\hat\Sigma \quad\checkmark \]
5. \[ \text{Generalised variance} = \det\hat\Sigma = \lambda_1\lambda_2\lambda_3 = 0 \]

1. \[ r_{13} = \frac{0.5}{\sqrt{0.5 \cdot 0.5}} = 1 \]
2. Attribute 3 is an exact linear function of attribute 1 (here \(x_3 = x_1 - 1\)). The data lies in a plane inside \(\mathbb R^3\), so there is no variance along \(\tfrac{1}{\sqrt2}(1, 0, -1)\) and the covariance matrix is singular.
3. Two components capture \[ \frac{2 + 1}{3} = 100\% \] The third column adds a dimension of storage but no information.

A singular \(\hat\Sigma\) has no inverse, so the Mahalanobis distance is undefined, and in regression a predictor that is an exact combination of others makes \(\tilde{\mathbf D}^T\tilde{\mathbf D}\) non-invertible (Lecture 7 slide 24). PCA handles it gracefully: it just reports \(\lambda = 0\).

1. \[ \boldsymbol\mu = \tfrac14(20, 20) = (5, 5) \]
2. \[ \mathbf Z = \begin{bmatrix} -4 & -1 \\ 0 & -1 \\ 2 & -1 \\ 2 & 3 \end{bmatrix} \]

1. \[ \mathbf Z^T\mathbf Z = \begin{bmatrix} 16 + 0 + 4 + 4 & 4 + 0 - 2 + 6 \\ 8 & 1 + 1 + 1 + 9 \end{bmatrix} = \begin{bmatrix} 24 & 8 \\ 8 & 12 \end{bmatrix} \]
2. \[ \Sigma = \begin{bmatrix} 6 & 2 \\ 2 & 3 \end{bmatrix} \]

1. \[ \operatorname{tr} = 9, \qquad \det = 18 - 4 = 14 \]
2. \[ \lambda^2 - 9\lambda + 14 = 0 \;\Rightarrow\; (\lambda - 7)(\lambda - 2) = 0 \]
3. \[ \lambda_1 = 7, \qquad \lambda_2 = 2 \]
4. \((\Sigma - 7I)\mathbf v = 0\): \(-v_1 + 2v_2 = 0\): \[ \mathbf v_1 = \tfrac{1}{\sqrt5}\begin{bmatrix} 2 \\ 1 \end{bmatrix} \approx \begin{bmatrix} 0.894 \\ 0.447 \end{bmatrix} \]
5. \[ \mathbf v_2 = \tfrac{1}{\sqrt5}\begin{bmatrix} -1 \\ 2 \end{bmatrix} \]

Always check the eigenvector in the second row too: \(2(2) - 4(1) = 0\) ✓. A sign slip in the first row gives a vector that fails the second.

1. \[ \frac{\lambda_1}{\lambda_1 + \lambda_2} = \frac{7}{9} \approx 0.778 \ge 0.75 \]
2. \(r = 1\): the smallest \(r\) whose cumulative fraction reaches \(\alpha\). With \(\alpha = 0.8\) it would be \(r = 2\).

1. \[ y_i = \frac{2z_{i1} + z_{i2}}{\sqrt5} \]
2. \[ \mathbf y = \tfrac{1}{\sqrt5}(-9,\ -1,\ 3,\ 7) = (-4.025,\ -0.447,\ 1.342,\ 3.130) \]
3. \[ \mathrm{Var}(y) = \frac{81 + 1 + 9 + 49}{5 \cdot 4} = \frac{140}{20} = 7 = \lambda_1 \quad\checkmark \]

1. \[ \tilde{\mathbf x}_i = \boldsymbol\mu + y_i\mathbf v_1 = (5,5) + \frac{2z_{i1} + z_{i2}}{5}\,(2, 1) \]
2. Multipliers \(\tfrac{-9}{5}, \tfrac{-1}{5}, \tfrac{3}{5}, \tfrac{7}{5}\): \[ \tilde{\mathbf x}: \quad (1.4, 3.2),\ (4.6, 4.8),\ (6.2, 5.6),\ (7.8, 6.4) \]
3. Errors: \[ (-0.4, 0.8),\ (0.4, -0.8),\ (0.8, -1.6),\ (-0.8, 1.6) \]
4. Squared lengths \(0.8, 0.8, 3.2, 3.2\): \[ \mathrm{MSE} = \tfrac{8}{4} = 2 = \lambda_2 \quad\checkmark \]

Each error vector is a multiple of \((-1, 2)\), the discarded direction. That is the geometry of Lecture 4 slide 14: the error is the projection onto the remaining basis vectors, orthogonal to the reconstruction.

1. Dot products: \[ \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 2 \end{bmatrix} \]
2. Square entrywise: \[ \mathbf K = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 4 \end{bmatrix} \]

1. \[ \phi(\mathbf x_1) = (1, 0, 0), \quad \phi(\mathbf x_2) = (0, 1, 0), \quad \phi(\mathbf x_3) = (1, 1, \sqrt2) \]
2. \[ \|\phi(\mathbf x_3)\|^2 = 1 + 1 + 2 = 4 = K_{33} \quad\checkmark \]

1. \[ \|\boldsymbol\mu_\phi\|^2 = \frac{1}{9}\sum_{i,j}K_{ij} = \frac{10}{9} \]
2. \[ d_\phi(\mathbf x_1, \mathbf x_2) = \sqrt{1 + 1 - 0} = \sqrt2 \]
3. \[ d_\phi(\mathbf x_1, \mathbf x_3) = \sqrt{1 + 4 - 2} = \sqrt3 \]
4. \[ \cos\theta_\phi = \frac{1}{\sqrt{1 \cdot 4}} = 0.5 \]

1. Row means \(\bar r = (\tfrac23, \tfrac23, 2)\), grand mean \(\bar g = \tfrac{10}{9}\).
2. \[ \hat K_{11} = 1 - \tfrac23 - \tfrac23 + \tfrac{10}{9} = \tfrac{9 - 12 + 10}{9} = \tfrac79 \]
3. \[ \hat K_{12} = 0 - \tfrac43 + \tfrac{10}{9} = -\tfrac29 \]
4. \[ \hat K_{13} = 1 - \tfrac23 - 2 + \tfrac{10}{9} = -\tfrac59 \]
5. \[ \hat K_{33} = 4 - 4 + \tfrac{10}{9} = \tfrac{10}{9} \]
6. \[ \hat{\mathbf K} = \tfrac19\begin{bmatrix} 7 & -2 & -5 \\ -2 & 7 & -5 \\ -5 & -5 & 10 \end{bmatrix} \]

Each row sums to zero ✓. By symmetry \(\hat K_{22} = \hat K_{11}\) and \(\hat K_{23} = \hat K_{13}\).

1. Work with \(9\hat{\mathbf K}\) and divide at the end. \[ 9\hat{\mathbf K}\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix} = \begin{bmatrix} 9 \\ -9 \\ 0 \end{bmatrix} \;\Rightarrow\; \eta = \tfrac99 = 1 \]
2. \[ 9\hat{\mathbf K}\begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 15 \\ 15 \\ -30 \end{bmatrix} \;\Rightarrow\; \eta = \tfrac{15}{9} = \tfrac53 \]
3. \[ 9\hat{\mathbf K}\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \mathbf 0 \;\Rightarrow\; \eta = 0 \]
4. Ordered: \[ \eta_1 = \tfrac53,\ \mathbf v_1 = \tfrac{1}{\sqrt6}(1, 1, -2); \quad \eta_2 = 1,\ \mathbf v_2 = \tfrac{1}{\sqrt2}(1, -1, 0); \quad \eta_3 = 0,\ \mathbf v_3 = \tfrac{1}{\sqrt3}(1, 1, 1) \]

The rows of a centred kernel matrix sum to zero, so \(\hat{\mathbf K}\mathbf 1 = \mathbf 0\): the all-ones vector is always an eigenvector with eigenvalue 0. That is why at most \(n - 1\) kernel components can be nonzero.

1. \[ \lambda_1 = \tfrac{5}{9} \approx 0.556, \qquad \lambda_2 = \tfrac13, \qquad \lambda_3 = 0 \]
2. \[ \frac{\eta_1}{\eta_1 + \eta_2} = \frac{5/3}{8/3} = 62.5\% \]
3. Coordinates \(\mathbf a_1 = \sqrt{\eta_1}\,\mathbf v_1\): \[ \sqrt{\tfrac53}\cdot\tfrac{1}{\sqrt6}(1, 1, -2) = \sqrt{\tfrac{5}{18}}\,(1, 1, -2) \approx (0.527,\ 0.527,\ -1.054) \]
4. Check: \[ \tfrac13(0.278 + 0.278 + 1.111) = 0.556 = \lambda_1 \quad\checkmark \]

On the second component the coordinates are \(\sqrt1 \cdot \tfrac{1}{\sqrt2}(1, -1, 0) = (0.707, -0.707, 0)\). PC 1 separates \(\mathbf x_3\) from the pair; PC 2 separates \(\mathbf x_1\) from \(\mathbf x_2\), just as in Lecture 6 slide 28.

1. \[ \mathbf S(1, 0, -1)^T = (1, 0, -1)^T \;\Rightarrow\; \lambda = 1 \]
2. For \((a, b, a)\): \(a + 0.9b = \lambda a\) and \(1.8a + b = \lambda b\): \[ (\lambda - 1)^2 = 1.62 \;\Rightarrow\; \lambda = 1 \pm 1.273 \]
3. \[ \lambda = 2.273 \text{ or } -0.273 \]
4. A negative eigenvalue means \(\mathbf S\) is not positive semidefinite, so no feature map produces it: it is not a valid kernel. Kernel PCA on it would report a negative variance.

Quick alternative: with unit diagonal, \(d_{AB} = d_{BC} = \sqrt{0.2} = 0.447\) but \(d_{AC} = \sqrt2 = 1.414 \gt 0.894\), which breaks the triangle inequality.

1. \[ \sum x_i = 10, \quad \sum x_i^2 = 30, \quad \sum y_i = 16, \quad \sum x_iy_i = 2 + 6 + 15 + 24 = 47 \]
2. \[ \mathbf A^T\mathbf A = \begin{bmatrix} 4 & 10 \\ 10 & 30 \end{bmatrix}, \qquad \mathbf A^T\mathbf y = \begin{bmatrix} 16 \\ 47 \end{bmatrix} \]

1. \[ \det = 120 - 100 = 20, \qquad (\mathbf A^T\mathbf A)^{-1} = \tfrac{1}{20}\begin{bmatrix} 30 & -10 \\ -10 & 4 \end{bmatrix} \]
2. \[ \begin{bmatrix} b \\ w \end{bmatrix} = \tfrac{1}{20}\begin{bmatrix} 480 - 470 \\ -160 + 188 \end{bmatrix} = \begin{bmatrix} 0.5 \\ 1.4 \end{bmatrix} \]
3. \[ \hat y = 0.5 + 1.4x \]
4. Check by the covariance route: \(S_{xx} = 5\), \(S_{xy} = 7\), \(w = 7/5 = 1.4\), \(b = 4 - 1.4(2.5) = 0.5\) ✓

1. \[ \hat{\mathbf y} = (1.9, 3.3, 4.7, 6.1)^T, \qquad \boldsymbol\epsilon = (0.1, -0.3, 0.3, -0.1)^T \]
2. \[ \boldsymbol\epsilon^T\mathbf 1 = 0.1 - 0.3 + 0.3 - 0.1 = 0 \quad\checkmark \]
3. \[ \boldsymbol\epsilon^T\mathbf x = 0.1 - 0.6 + 0.9 - 0.4 = 0 \quad\checkmark \]
4. \[ \mathrm{SSE} = 0.01 + 0.09 + 0.09 + 0.01 = 0.2, \qquad \mathrm{RMSE} = \sqrt{0.2 / 4} \approx 0.224 \]

\(\hat{\mathbf y}\) is the orthogonal projection of \(\mathbf y\) onto \(\mathrm{span}\{\mathbf 1, \mathbf x\}\): the residual is perpendicular to the plane, so no other line can have a smaller SSE.

In high dimensions almost all of a hypersphere's surface is far from any one fixed direction: a random unit vector is nearly orthogonal to \(\mathbf v_1\) with overwhelming probability, and the fraction of directions within a small angle of \(\mathbf v_1\) shrinks exponentially with \(m\). The eigen-decomposition does not search; it solves \(\Sigma\mathbf v = \lambda\mathbf v\) exactly.

No. Adding any predictor can never increase the training SSE, because the old fit is still available with weight 0 on the new column and least squares can only do at least as well. With \(d + 1 = n\) parameters the fit is perfect regardless of meaning. Only the error on new data says whether a predictor helps.