Data, vectors and distributions

Lecture 1 (Data Matrix: Vector View) and Lecture 2 (Data Matrix: Probabilistic View), rebuilt as one storyline: first the data as a table of vectors, then one column of that table as a random variable.

How to read this page. It is ordered by concept, not by slide. Every card says which slides it came from, what to read first, and what uses it later. These two lectures are the toolbox that Lectures 3–4 (covariance to PCA) builds on.

Teal = explanation added by Claude, not on the slides. Amber = watch out. Dropdowns hide worked examples, practice questions, and their answers.

The road through the two lectures

Data matrix→Norm→Inner product & angle→Projection→Distance→Random variable→pmf→cdf→Mean→Variance

Attribute types, linear independence, the median and the IQR are side stops: they appear in the lectures and homework, but the road to PCA does not need them.

Lecture map: which slides each card covers

Part 1

Data as a table

The data matrix: objects and attributes

A dataset is a table. Each row is one thing you measured (a student, a flower) and each column is one measurement (GPA, petal length).

Data can often be represented as an \(n \times m\) matrix of attribute values.

  • Rows: entities, aka instances, samples, points, feature vectors, records, objects, tuples.
  • Columns: attributes, aka features, properties, fields, variables, dimensions.
  • Dimensionality of the data = number of columns = \(m\).

The \(i\)-th feature vector is the \(i\)-th row, written as a column vector:

\[ \mathbf x_i = \begin{bmatrix} x_{i1} \\ x_{i2} \\ \vdots \\ x_{im} \end{bmatrix} = (x_{i1},\ x_{i2},\ \dots,\ x_{im})^T \]

Example: the transcript of the first student is the vector of their course GPAs.

\(n\)
number of rows (objects, samples)
\(m\)
number of columns (attributes), also called \(d\) in later lectures
\(x_{ij}\)
value of attribute \(j\) for object \(i\)
\(\mathbf x_i\)
row \(i\) as a column vector in \(\mathbb R^m\)
\(^T\)
transpose: turns a row into a column and back

L1 s6 uses bold \(\mathbf x_i\) for row \(i\) (one object) and italic \(\boldsymbol x_j\) for column \(j\) (one attribute). Later lectures (L3 s8, L5 s2, L6 s13) switch between the two. Before any calculation, write down which one you are holding and whether the points are rows or columns.

Running example: the Iris dataset

  • 3 classes: Setosa, Versicolor, Virginica, 50 samples each, so \(n = 150\).
  • 4 attributes (\(m = 4\)): sepal length, sepal width, petal length, petal width, all in cm.
  • Each row holds a feature vector plus the class label in the last column.
\[ \mathbf x_1 = (5.1,\ 3.5,\ 1.4,\ 0.2)^T,\ \text{class Iris-setosa} \]

Every lecture uses Iris for its examples, so remember the shape: 150 rows, 4 numeric columns, one label column that you drop before doing any algebra.

Attribute types: numeric and categorical

Before you compute anything, check what kind of values a column holds. You can add and compare numbers, but you cannot add labels like "male" or "PhD".

Numeric attributes have real (\(x_{ij} \in \mathbb R\)) or integer (\(x_{ij} \in \mathbb Z\)) values: continuous or discrete. A binary attribute has \(x_{ij} \in \{0,1\}\).

  • Some are discrete by nature (number of kids, sick days). Others are made discrete by discretising a continuous one.
  • GPA in \([0,4]\) is continuous. Age is continuous, but counted in years it is discrete.

Interval-scaled: only differences make sense, not ratios. Temperature: a drop of 10° is meaningful, but "twice as cold" is not.

Ratio-scaled: differences and ratios both make sense. Age: 24 is twice 12, and A is 2 years older than B.

Categorical attributes take values in a set of symbols \(S = \{a_0, a_1, \dots, a_{K-1}\}\).

  • Nominal: unordered, only equality can be tested. Gender: \(S = \{M, F\}\).
  • Ordinal: ordered, equality and inequality can be tested. Education: HS \(\lt\) BS \(\lt\) MS \(\lt\) PhD.

N.B. we cannot perform algebraic operations with categorical variables.

AskIf yesIf no
Is it a number you can add?numericcategorical
Numeric: can it take any value in a range?continuousdiscrete (periodic, binary, counts)
Numeric: is zero a true "nothing", so ratios make sense?ratio-scaledinterval-scaled
Categorical: do the values have an order?ordinalnominal

Nominal values can only be tested with \(=\). Writing "male \(\gt\) female" is meaningless. Ordinal values allow \(\lt\) and \(\gt\) but still no arithmetic: PhD \(-\) BS is not a number.

Part 2

Rows as vectors: length, angle, projection, distance

The algebra every later lecture reuses. Lectures 3–4 call it "the algebraic view".

Norm: the length of a vector

The norm is the length of the arrow from the origin to the point. The usual one is the Euclidean norm, which is Pythagoras in \(m\) dimensions.

A vector is characterised by its length and direction. The length is measured by its Euclidean norm:

\[ \|\mathbf x\|_2 = \sqrt{\mathbf x^T \mathbf x} \] \[ \|\mathbf x\|_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_m^2} \]

Scaling by a number \(\alpha\):

\[ \|\alpha\,\mathbf x\| = |\alpha| \cdot \|\mathbf x\| \]

Normalisation. A vector \(\mathbf x\) is normalised if \(\|\mathbf x\| = 1\). To normalise any vector, divide by its norm:

\[ \tilde{\mathbf x} = \frac{\mathbf x}{\|\mathbf x\|} \]

Definition (L1 s19). A norm assigns a non-negative number \(\|\mathbf x\|\) to each vector and satisfies:

  1. Positivity: \(\|\mathbf x\| \ge 0\), and \(\|\mathbf x\| = 0\) only if \(\mathbf x = \mathbf 0\).
  2. Scaling: \(\|\lambda \mathbf x\| = |\lambda|\,\|\mathbf x\|\).
  3. Triangle inequality: \(\|\mathbf x + \mathbf y\| \le \|\mathbf x\| + \|\mathbf y\|\).
\(\|\mathbf x\|\), \(\|\mathbf x\|_2\)
Euclidean length (the subscript 2 is often dropped)
\(\mathbf x^T\mathbf x\)
the vector dotted with itself, equal to the sum of squares
\(\tilde{\mathbf x}\), \(\mathbf u_x\)
the unit vector pointing the same way as \(\mathbf x\)

Recipe: normalise a vector

  1. Square every component and add them up.
  2. Take the square root. That is the norm.
  3. Divide every component by the norm. Check: the result has norm 1.

The p-norm family: \(L_1\), \(L_2\), \(L_\infty\)

The \(p\)-norm (for \(p \ge 1\)), also called the \(L_p\) norm:

\[ \|\mathbf x\|_p = \Big( |x_1|^p + |x_2|^p + \dots + |x_m|^p \Big)^{1/p} \]

\(L_1\)-norm (\(p = 1\)):

\[ \|\mathbf x\|_1 = |x_1| + |x_2| + \dots + |x_m| \]

\(L_2\)-norm (\(p = 2\)): the Euclidean norm.

\(L_\infty\)-norm (sup-norm):

\[ \|\mathbf x\|_\infty = \max\{|x_1|, |x_2|, \dots, |x_m|\} \]

For any vector:

\[ \|\mathbf x\|_\infty \le \|\mathbf x\|_2 \le \|\mathbf x\|_1 \]

\(L_1\) is the walking distance along city blocks, \(L_2\) is the straight-line distance, \(L_\infty\) only counts the single biggest step. Changing the norm can change which point is "closest" (see the kNN example under Distance).

The three norms of \(\mathbf x = (1, -2)^T\)
  1. \[ \|\mathbf x\|_1 = |1| + |-2| = 3 \]
  2. \[ \|\mathbf x\|_2 = \sqrt{1^2 + (-2)^2} = \sqrt 5 \approx 2.236 \]
  3. \[ \|\mathbf x\|_\infty = \max\{1, 2\} = 2 \]
  4. Check the ordering: \(2 \le 2.236 \le 3\) ✓
Norms and unit vectors for two Iris directions

\(\mathbf v_1 = (3.0, 1.1)^T\) and \(\mathbf v_2 = (5.6, 1.4)^T\).

  1. \[ \|\mathbf v_1\|_2 = \sqrt{3.0^2 + 1.1^2} = \sqrt{10.21} = 3.1953 \]
  2. \[ \|\mathbf v_2\|_2 = \sqrt{5.6^2 + 1.4^2} = \sqrt{33.32} = 5.7723 \]
  3. \[ \tilde{\mathbf v}_1 = \frac{\mathbf v_1}{3.1953} = (0.939,\ 0.344)^T \]
  4. \[ \tilde{\mathbf v}_2 = \frac{\mathbf v_2}{5.7723} = (0.970,\ 0.243)^T \]

L1 s17 prints \(\|\mathbf v_2\| = 5.7728\) and "31953" for 3.1953. The correct value is \(\sqrt{33.32} = 5.7723\); the rest of the slide's numbers are unaffected.

Magnitude of two 5-dimensional vectors

\(V_1 = (1, 2, 3, 4, 5)^T\), \(V_2 = (5, -4, 3, -2, 1)^T\). Calculate the magnitude of each.

  1. \[ \|V_1\| = \sqrt{1 + 4 + 9 + 16 + 25} = \sqrt{55} = 7.4162 \]
  2. \[ \|V_2\| = \sqrt{25 + 16 + 9 + 4 + 1} = \sqrt{55} = 7.4162 \]
  3. Both vectors have the same length.

Inner product and the angle between vectors

The inner product multiplies two vectors position by position and adds up the results. It is large when the vectors point the same way, and it gives you the angle between them.

The inner product, aka scalar product or dot product, of \(\mathbf x, \mathbf y \in \mathbb R^m\):

\[ \langle \mathbf x, \mathbf y \rangle = \mathbf x \cdot \mathbf y = \mathbf y^T \mathbf x = \sum_{i=1}^m x_i y_i \] \[ \langle \mathbf x, \mathbf y \rangle = x_1 y_1 + x_2 y_2 + \dots + x_m y_m \]

It can also be expressed with the smallest angle \(\theta\) between the two vectors:

\[ \langle \mathbf x, \mathbf y \rangle = \mathbf x^T \mathbf y = \|\mathbf x\|\,\|\mathbf y\| \cos\theta \]

So the angle is

\[ \theta = \angle(\mathbf x, \mathbf y) = \cos^{-1}\!\left( \frac{\mathbf x^T \mathbf y}{\|\mathbf x\| \cdot \|\mathbf y\|} \right) \]
\(\langle \mathbf x, \mathbf y \rangle\), \(\mathbf x^T\mathbf y\)
three notations for the same number
\(\cos\theta\)
cosine similarity: \(+1\) same direction, \(0\) perpendicular, \(-1\) opposite
\(\cos^{-1}\)
arccos; gives the angle in radians, convert to degrees if asked

Recipe: angle between two vectors

  1. Compute the dot product.
  2. Compute both norms.
  3. \[ \cos\theta = \frac{\text{dot product}}{\text{norm} \times \text{norm}} \]
  4. \[ \theta = \cos^{-1}(\cos\theta) \] Multiply by \(180/\pi\) for degrees.

The dot product only equals \(\|\mathbf x\|^2\) when \(\mathbf y = \mathbf x\). Mixing up \(\mathbf x^T\mathbf y\) (a number) with \(\mathbf x\mathbf y^T\) (an \(m \times m\) matrix, used in covariance later) is a common slip.

Dot product and angle of \(\mathbf v_1\) and \(\mathbf v_2\)
  1. \[ \langle \mathbf v_1, \mathbf v_2 \rangle = 3.0 \times 5.6 + 1.1 \times 1.4 = 18.34 \]
  2. \[ \cos\theta = \frac{18.34}{3.1953 \times 5.7723} = 0.9943 \]
  3. \[ \theta = \cos^{-1}(0.9943) = 6.1^\circ \]
  4. The two directions are almost parallel, which matches the Iris plot on the slide.
Semantic search: which document is closest to the query?

Vocabulary (neural, network, recipe). Query \(\mathbf q = (1,1,0)\), documents \(\mathbf d_1 = (2,2,0)\), \(\mathbf d_2 = (6,6,1)\), \(\mathbf d_3 = (0,1,5)\). What should "best match" mean numerically?

  1. Dot products: \[ \mathbf q^T\mathbf d_1 = 2 + 2 + 0 = 4 \]
  2. \[ \mathbf q^T\mathbf d_2 = 6 + 6 + 0 = 12 \]
  3. \[ \mathbf q^T\mathbf d_3 = 0 + 1 + 0 = 1 \]
  4. Norms: \[ \|\mathbf q\| = \sqrt 2,\quad \|\mathbf d_1\| = \sqrt 8 \]
  5. \[ \|\mathbf d_2\| = \sqrt{73},\quad \|\mathbf d_3\| = \sqrt{26} \]
  6. Cosines: \[ \cos\theta_1 = \frac{4}{\sqrt 2 \sqrt 8} = 1.000 \]
  7. \[ \cos\theta_2 = \frac{12}{\sqrt 2 \sqrt{73}} = 0.993 \]
  8. \[ \cos\theta_3 = \frac{1}{\sqrt 2 \sqrt{26}} = 0.139 \]
  9. Euclidean distances: \[ \|\mathbf q - \mathbf d_1\| = \sqrt 2 = 1.414 \]
  10. \[ \|\mathbf q - \mathbf d_2\| = \sqrt{51} = 7.141 \]
  11. \[ \|\mathbf q - \mathbf d_3\| = \sqrt{26} = 5.099 \]
  12. Reading: the raw dot product picks \(\mathbf d_2\) only because it is long (more words). Cosine says \(\mathbf d_1\) points exactly the same way as the query, and \(\mathbf d_2\) almost does. Distance punishes \(\mathbf d_2\) for its length. For text, cosine similarity is the usual choice because it ignores document length.
Dot product, angle and cosine similarity in 5-D

\(V_1 = (1, 2, 3, 4, 5)^T\), \(V_2 = (5, -4, 3, -2, 1)^T\). (a) Dot product. (c) Angle in degrees and cosine similarity. (d) Relation between the vectors.

  1. \[ V_1^T V_2 = 5 - 8 + 9 - 8 + 5 = 3 \]
  2. From part (b): \(\|V_1\| = \|V_2\| = \sqrt{55} = 7.4162\).
  3. \[ \cos\theta = \frac{3}{7.4162 \times 7.4162} = \frac{3}{55} = 0.0545 \]
  4. \[ \theta = \cos^{-1}(0.0545) = 86.87^\circ \]
  5. (d) Cosine near 0 and angle near \(90^\circ\): the vectors are nearly orthogonal, and they have the same length.
Read two vectors off a grid, then cosine and angle

From the grid: \(\mathbf u = (3, 4)\), \(\mathbf v = (3, 0)\). Compute the cosine similarity and the angle in degrees.

  1. \[ \mathbf u \cdot \mathbf v = 3 \times 3 + 4 \times 0 = 9 \]
  2. \[ \|\mathbf u\| = \sqrt{9 + 16} = 5 \]
  3. \[ \|\mathbf v\| = \sqrt{9 + 0} = 3 \]
  4. \[ \cos\theta = \frac{9}{5 \times 3} = 0.6 \]
  5. \[ \theta = \cos^{-1}(0.6) \approx 53.1^\circ \]

Orthogonal and orthonormal vectors

Two vectors are orthogonal (at right angles) when their dot product is zero. An orthonormal set is a set of unit vectors that are all orthogonal to each other.

Two vectors \(\mathbf x\) and \(\mathbf y\) are orthogonal (or normal) if

\[ \mathbf x^T \mathbf y = 0 \]

Example: \(\mathbf x = (1, -2, 2)\) and \(\mathbf y = (2, -1, -2)\).

An orthonormal set \(\{\mathbf x_1, \dots, \mathbf x_n\}\) is a set of unit-norm, mutually orthogonal vectors:

\[ \mathbf x_i^T \mathbf x_j = \delta_{ij} = \begin{cases} 1, & i = j \\ 0, & i \ne j \end{cases} \]

where \(\delta_{ij}\) is the Kronecker delta function.

The eigenvectors of a covariance matrix form an orthonormal set. That is what lets PCA treat each principal direction separately (see eigen-decomposition).

Checking that \((1,-2,2)\) and \((2,-1,-2)\) are orthogonal
  1. \[ \mathbf x^T\mathbf y = 1 \times 2 + (-2)(-1) + 2 \times (-2) \]
  2. \[ \mathbf x^T\mathbf y = 2 + 2 - 4 = 0 \] Orthogonal ✓
Which pairs are orthogonal?

(a) \((2,1,1)\), \((8,2,2)\). (b) \((1,2,3)\), \((2,-2,-3)\). (c) \((1,2,4)\), \((2,1,3)\).

  1. \[ \text{(a)}\quad 16 + 2 + 2 = 20 \ne 0 \]
  2. \[ \text{(b)}\quad 2 - 4 - 9 = -11 \ne 0 \]
  3. \[ \text{(c)}\quad 2 + 2 + 12 = 16 \ne 0 \]
  4. None of the pairs is orthogonal. (In (a) the second vector is 4 times the first: they are parallel, the opposite of orthogonal.)

Orthogonal projection of one vector onto another

Projecting \(\mathbf y\) onto \(\mathbf x\) means finding the shadow of \(\mathbf y\) along the line of \(\mathbf x\). What is left over is at right angles to \(\mathbf x\).

The projection of a vector \(\mathbf y\) onto a vector \(\mathbf x\) is a vector:

\[ \mathbf y_\parallel = \|\mathbf y\| \cos\theta\ \mathbf u_x = \frac{\langle \mathbf x, \mathbf y \rangle}{\|\mathbf x\|}\,\mathbf u_x \]

where \(\mathbf u_x\) is the unit vector along \(\mathbf x\):

\[ \mathbf u_x = \frac{\mathbf x}{\|\mathbf x\|} \]

Note that

\[ \mathbf y = \mathbf y_\parallel + \mathbf y_\perp \]
y split into a part along x and a part perpendicular to x
L1 s16: \(\mathbf y\) splits into \(\mathbf y_\parallel\) along \(\mathbf x\) and \(\mathbf y_\perp\) at right angles to it.
\(\mathbf y_\parallel\)
the part of \(\mathbf y\) along \(\mathbf x\) (a vector)
\(\mathbf y_\perp\)
the part of \(\mathbf y\) at right angles to \(\mathbf x\): \(\mathbf y - \mathbf y_\parallel\)
\(\|\mathbf y\|\cos\theta\)
the scalar projection: the length of the shadow, with a sign
\[ \mathbf y_\parallel = \frac{\mathbf x^T \mathbf y}{\mathbf x^T \mathbf x}\,\mathbf x \]

Divide the dot product by \(\|\mathbf x\|^2\), then scale \(\mathbf x\). This is the form the HW1 key and Quiz 1 use. If \(\mathbf x\) is already a unit vector, the shadow length is just \(\mathbf x^T\mathbf y\), which is how PCA computes coordinates (projection onto a basis).

Recipe: project \(\mathbf y\) onto \(\mathbf x\)

  1. \[ s = \frac{\mathbf x^T\mathbf y}{\mathbf x^T\mathbf x} \]
  2. \[ \mathbf y_\parallel = s\,\mathbf x \]
  3. \[ \mathbf y_\perp = \mathbf y - \mathbf y_\parallel \]
  4. Check: \(\mathbf x^T\mathbf y_\perp = 0\).

"Project \(\mathbf y\) onto \(\mathbf x\)" and "project \(\mathbf x\) onto \(\mathbf y\)" give different vectors: the result always points along the second one. Read the question twice.

Projecting \(\mathbf v_1\) onto \(\mathbf v_2\), the full calculation

\(\mathbf v_1 = (3.0, 1.1)^T\), \(\mathbf v_2 = (5.6, 1.4)^T\).

  1. \[ \mathbf u_1 = \frac{\mathbf v_1}{3.1953} = (0.9389,\ 0.3443)^T \]
  2. \[ \mathbf u_2 = \frac{\mathbf v_2}{5.7723} = (0.9701,\ 0.2425)^T \]
  3. \[ \mathbf v_1^T\mathbf v_2 = 18.34 \]
  4. \[ \cos\theta = \frac{18.34}{3.1953 \times 5.7723} = 0.9943 \]
  5. \[ \theta = \cos^{-1}(0.9943) = 6.1^\circ \]
  6. Scalar projection: \[ \|\mathbf v_1\|\cos\theta = 3.1953 \times 0.9943 = 3.1772 \]
  7. \[ \mathbf v_{1\parallel} = 3.1772\,\mathbf u_2 = (3.0824,\ 0.7706)^T \]
  8. \[ \mathbf v_{1\perp} = \mathbf v_1 - \mathbf v_{1\parallel} = (-0.0824,\ 0.3294)^T \]
  9. \[ \|\mathbf v_{1\perp}\| = \|\mathbf v_1\|\sin\theta = 0.3395 \]
Debiasing word embeddings: remove the component along one direction

Gender direction \(\mathbf g = (3,4)\), \(\|\mathbf g\| = 5\). Occupation vectors \(\mathbf x_{\text{nurse}} = (2,6)\), \(\mathbf x_{\text{eng}} = (0,-5)\). How much of each lies along \(\mathbf g\), and what survives once it is removed?

  1. Unit direction: \[ \mathbf u_g = \frac{\mathbf g}{5} = (0.6,\ 0.8) \]
  2. Nurse, scalar projection: \[ \frac{\mathbf x^T\mathbf g}{\|\mathbf g\|} = \frac{6 + 24}{5} = 6 \]
  3. Nurse, component along \(\mathbf g\): \[ \mathbf x_\parallel = 6\,\mathbf u_g = (3.6,\ 4.8) \]
  4. Nurse, what survives: \[ \mathbf x_\perp = (2,6) - (3.6, 4.8) = (-1.6,\ 1.2) \] Check: \((-1.6)(3) + (1.2)(4) = 0\) ✓
  5. Engineer, scalar projection: \[ \frac{0 - 20}{5} = -4 \]
  6. Engineer, component along \(\mathbf g\): \[ \mathbf x_\parallel = -4\,\mathbf u_g = (-2.4,\ -3.2) \]
  7. Engineer, what survives: \[ \mathbf x_\perp = (0,-5) - (-2.4,-3.2) = (2.4,\ -1.8) \]
  8. Reading: "nurse" leans \(+6\) along the gender direction, "engineer" leans \(-4\). Debiasing keeps only \(\mathbf x_\perp\), so both words now have zero component along \(\mathbf g\) and everything else is untouched.
Projection of \(\mathbf u = (3,4)\) onto \(\mathbf v = (3,0)\)
  1. \[ s = \frac{\mathbf u\cdot\mathbf v}{\|\mathbf v\|^2} = \frac{9}{9} = 1 \]
  2. \[ \operatorname{proj}_{\mathbf v}(\mathbf u) = 1 \times (3, 0) = (3,\ 0) \]
  3. Makes sense: \(\mathbf v\) lies on the x-axis, so the shadow of \((3,4)\) is just its x-part.
Projection of \(V_1\) onto \(V_2\) in 5-D

\(V_1 = (1, 2, 3, 4, 5)^T\), \(V_2 = (5, -4, 3, -2, 1)^T\).

  1. \[ s = \frac{V_1^T V_2}{V_2^T V_2} = \frac{3}{55} = 0.0545 \]
  2. \[ \operatorname{proj}_{V_2}(V_1) = 0.0545 \times (5, -4, 3, -2, 1)^T \]
  3. \[ \operatorname{proj}_{V_2}(V_1) = (0.2727,\ -0.2182,\ 0.1636,\ -0.1091,\ 0.0545)^T \]

Distance between two vectors

The distance between two points is the length of the arrow from one to the other. Which norm you use decides what "length" means.

The distance between \(\mathbf x_1\) and \(\mathbf x_2\) is the norm of the difference vector \(\mathbf v = \mathbf x_1 - \mathbf x_2\):

\[ d(\mathbf x_1, \mathbf x_2) = \|\mathbf x_1 - \mathbf x_2\| \]

Euclidean distance uses the \(L_2\) norm; city-block (Manhattan) distance uses the \(L_1\) norm.

Recipe

  1. Subtract the two vectors component by component.
  2. Apply the norm the question asks for (\(L_2\) if it does not say).

Lecture 5 writes the same squared distance as a dot product: \(\|\mathbf x - \mathbf y\|^2 = \mathbf x^T\mathbf x + \mathbf y^T\mathbf y - 2\mathbf x^T\mathbf y\). That identity is what lets kernels compute distances in feature space.

Euclidean and city-block distance between \(\mathbf v_1\) and \(\mathbf v_2\)
  1. \[ \mathbf v_1 - \mathbf v_2 = (3.0 - 5.6,\ 1.1 - 1.4)^T = (-2.6,\ -0.3)^T \]
  2. \[ d_2 = \sqrt{(2.6)^2 + (0.3)^2} = \sqrt{6.85} = 2.6173 \]
  3. \[ d_1 = |-2.6| + |-0.3| = 2.9 \]
k-nearest neighbour on Iris: does the norm change the answer?

Query \(\mathbf q = (4.5, 1.5)\) (petal length, petal width in cm). Candidates \(\mathbf a = (4.8, 1.5)\) [versicolor] and \(\mathbf b = (4.7, 1.7)\) [virginica]. Which is nearer under \(L_2\), \(L_1\), \(L_\infty\)? Does it change if we measure in mm?

  1. \[ \mathbf q - \mathbf a = (-0.3,\ 0) \]
  2. \[ \mathbf q - \mathbf b = (-0.2,\ -0.2) \]
  3. \(L_2\): \[ d(\mathbf q,\mathbf a) = 0.300 \]
  4. \[ d(\mathbf q,\mathbf b) = \sqrt{0.08} = 0.283 \] Nearest: \(\mathbf b\), virginica.
  5. \(L_1\): \[ d(\mathbf q,\mathbf a) = 0.3 \]
  6. \[ d(\mathbf q,\mathbf b) = 0.4 \] Nearest: \(\mathbf a\), versicolor.
  7. \(L_\infty\): \[ d(\mathbf q,\mathbf a) = 0.3 \]
  8. \[ d(\mathbf q,\mathbf b) = 0.2 \] Nearest: \(\mathbf b\), virginica.
  9. In mm every coordinate is multiplied by 10, so every distance is multiplied by 10 and the ranking is unchanged. Scaling only matters when the two attributes are scaled differently.
  10. Reading: the predicted class depends on the norm. \(L_1\) punishes \(\mathbf b\) for being off in two coordinates; \(L_\infty\) only looks at the worst coordinate.

Linear independence, span, basis and dimension

Vectors are independent when none of them can be built from the others. The number of independent directions you really have is the dimension, which can be smaller than the number of columns.

A set \(\{\mathbf x_1, \dots, \mathbf x_n\}\) in \(\mathbb R^m\) is linearly independent if there are no real numbers \(\alpha_1, \dots, \alpha_n\) such that

\[ \alpha_1 \mathbf x_1 + \alpha_2 \mathbf x_2 + \dots + \alpha_n \mathbf x_n = \mathbf 0 \]

unless \(\alpha_1 = \alpha_2 = \dots = \alpha_n = 0\).

  • This means none of the vectors can be expressed as a linear combination of the others.
  • A set of orthogonal vectors is linearly independent, but the converse is not necessarily true.

\(\{\mathbf v_1, \dots, \mathbf v_n\}\) is a spanning set for a subspace \(\mathcal V\) if any \(\mathbf x \in \mathcal V\) can be written as

\[ \mathbf x = \alpha_1 \mathbf v_1 + \alpha_2 \mathbf v_2 + \dots + \alpha_n \mathbf v_n \]

Linearly independent and spanning: a basis. The number of vectors in a basis is the dimension of \(\mathcal V\).

Put the vectors as columns of a matrix. If one column equals a combination of the others, the vectors are dependent and the rank (true dimension) is less than the number of columns. In PCA, dimension reduction is the same idea: find a small basis that still spans (almost) all of the data.

Redundant sensors: three columns, how many dimensions?

A logger stores intercept (all ones), °C and °F: rows \((1, 20, 68.0)\), \((1, 22, 71.6)\), \((1, 25, 77.0)\), \((1, 21, 69.8)\).

  1. Check the rule between the columns: \[ 1.8 \times 20 + 32 = 68.0 \]
  2. \[ 1.8 \times 22 + 32 = 71.6 \] It holds in every row.
  3. So the °F column is a linear combination of the other two: \[ \mathbf x_{^\circ F} = 1.8\,\mathbf x_{^\circ C} + 32\,\mathbf 1 \]
  4. In the definition: \(32\,\mathbf x_1 + 1.8\,\mathbf x_2 - 1\,\mathbf x_3 = \mathbf 0\) with non-zero coefficients, so the three columns are dependent.
  5. Any two of the columns span the same space as all three: the data has 3 columns but only 2 dimensions of information (rank 2). The third column adds nothing new and would make a covariance matrix singular.

Part 3

One attribute as a random variable

Lecture 2 looks at a single column of the data matrix and asks how likely each value is.

An attribute as a random variable

A random variable is a column of the table seen as "a number we did not know until we measured it". It is discrete if it can only take separate values, continuous if it can take any value in a range.

An attribute can be considered as a random variable (RV), denoted by \(X\). For Iris petal length, we have a population of 150 realisations (samples), and we ask: what is the probability of \(X \le x\)?

Definition. A random variable \(X\) is a function that assigns a real number \(x\) to each outcome of an experiment.

  • \(\operatorname{range}(X)\) may be a finite set of numbers or a continuous interval on \(\mathbb R\).
  • \(X\) is discrete if it takes only a finite (or countably infinite) set of values; continuous if its range is an interval in \(\mathbb R\).

Examples: a die roll, \(\operatorname{range}(X) = \{1, \dots, 6\}\); a coin toss, \(\{0, 1\}\); Iris petal length, \(x \in [1, 7]\), first five values \(\{1.4, 1.4, 1.3, 1.5, 1.4\}\).

\(X\)
capital: the random variable (the idea)
\(x\), \(x_i\)
lower case: one observed value
\(\Pr[\cdot]\)
probability of an event

Probability mass function (discrete)

For a discrete variable, the pmf is the list of probabilities, one for each possible value. They are each between 0 and 1 and they add up to 1.

For a discrete RV, let \(p(x)\) denote the probability that \(X\) has value \(x\):

\[ p(x) = \Pr[X = x] \]

If \(X \in \{x_1, \dots, x_n\}\), the probability mass function (pmf) is the set of values

\[ \text{pmf}: p(x) = \{p(x_1),\ p(x_2),\ \dots,\ p(x_n)\} \]

where

\[ 0 \le p(x_i) \le 1\ \ \forall i,\qquad \sum_{i=1}^n p(x_i) = 1 \]

A certain event has \(p(x) = 1\); the impossible event \(\phi\) has \(p(\phi) = 0\).

Fair coin toss, \(X \in \{0,1\}\):

\[ p(0) = p(1) = \tfrac12 \]

Die roll, \(X \in \{1,\dots,6\}\):

\[ p(x) = \tfrac16 \text{ for each } x \]

Always check the two conditions first. A question that gives you a pmf with an unknown constant is asking you to make the probabilities sum to 1.

Read a pmf and check it

\(p(1) = \frac1{10},\ p(2) = \frac3{10},\ p(3) = \frac2{10},\ p(4) = \frac3{10},\ p(5) = \frac1{10}\). Plot \(p(x)\).

  1. \[ \tfrac1{10} + \tfrac3{10} + \tfrac2{10} + \tfrac3{10} + \tfrac1{10} = 1 \] Valid pmf ✓
  2. The plot is five vertical stems at \(x = 1,\dots,5\) with heights 0.1, 0.3, 0.2, 0.3, 0.1, and zero everywhere else. It is symmetric about \(x = 3\).

Cumulative distribution function (discrete)

The cdf at \(x\) is the probability of getting \(x\) or less. For a discrete variable it is a staircase: add up the pmf values as you move right.

The cumulative distribution function (cdf) \(F(x)\) is defined by

\[ F(x) = \Pr[X \le x] = \sum_{i=1}^{k} p(x_i),\quad \text{such that } x_k \le x \]

Note that \(x\) can take any real value, and \(F(x)\) satisfies

\[ F(-\infty) = 0,\qquad 0 \le F(x) \le 1,\qquad F(\infty) = 1 \] \[ x_1 \le x_2 \implies F(x_1) \le F(x_2) \]

Example: cdf of a die roll,

\[ F(x) = \begin{cases} 0, & x \lt 1 \\ \sum_{i=1}^{\lfloor x \rfloor} p(i), & 1 \le x \le 6 \\ 1, & x \ge 6 \end{cases} \]

where \(\lfloor x \rfloor\) is the largest integer smaller than (or equal to) \(x\).

Probability of an interval from the cdf: \(\Pr(a \lt X \le b) = F(b) - F(a)\). For a discrete variable, be careful at the endpoints: \(\Pr(a \le X \le b)\) includes \(a\), so subtract \(F\) just below \(a\).

Iris petal length cut into three categories
  1. Short: \(X \le 2.5\). Medium: \(2.5 \lt X \le 5\). Long: \(X \gt 5\). Code them as \(X \in \{0, 1, 2\}\).
  2. Count the flowers in each bin: \[ \Pr(\text{Short}) = \tfrac{50}{150} = \tfrac13 \]
  3. \[ \Pr(\text{Medium}) = \tfrac{58}{150} = 0.3867 \]
  4. \[ \Pr(\text{Long}) = \tfrac{42}{150} = 0.28 \]
  5. The cdf steps up at 0, 1, 2: \[ F(0) = 0.3333,\quad F(1) = 0.72,\quad F(2) = 1 \]
Build the cdf from a pmf, then an interval probability

Same pmf as above: \(p(1..5) = 0.1, 0.3, 0.2, 0.3, 0.1\). (b) Compute and plot \(F(x)\). (c) Compute \(\Pr(1.4 \le X \le 4.2)\).

  1. Running sums: \[ F(1) = 0.1,\quad F(2) = 0.4,\quad F(3) = 0.6 \]
  2. \[ F(4) = 0.9,\quad F(5) = 1 \]
  3. \[ F(x) = \begin{cases} 0, & x \lt 1 \\ \sum_{i=1}^{\lfloor x\rfloor} p(i), & 1 \le x \lt 5 \\ 1, & x \ge 5 \end{cases} \] A staircase that jumps at 1, 2, 3, 4, 5.
  4. (c) The values inside \([1.4, 4.2]\) are 2, 3, 4: \[ \Pr(1.4 \le X \le 4.2) = F(4.2) - F(1.4) = F(4) - F(1) \]
  5. \[ \Pr = 0.9 - 0.1 = 0.8 \]

Continuous variables: cdf and probability density

A continuous variable has probability zero of hitting any exact value. Instead it has a density: a curve whose area over an interval is the probability of landing in that interval.

For a continuous RV \(X\), \(\Pr(X = x) = 0\). However \(\Pr(X \le x)\) is well defined for all \(x\):

\[ F(x) = \Pr(X \le x) \] \[ F(-\infty) = 0,\qquad 0 \le F(x) \le 1,\qquad F(\infty) = 1 \]

Probability density function (pdf): the derivative of the cdf.

\[ p(x) = \frac{dF(x)}{dx} \] \[ F(x) = \int_{-\infty}^{x} p(x)\,dx \] \[ \int_{-\infty}^{\infty} p(x)\,dx = 1 \] \[ p(x) \ge 0\ \ \forall x \in \mathbb R \] \[ \Pr(x_0 \lt X \le x_1) = \int_{x_0}^{x_1} p(x)\,dx \]

Interpretation (L2 s9): the pdf does not give the probability at \(x\):

\[ p(x) \ne \Pr(X = x) = 0 \]

It is a density, which must be integrated:

\[ \Pr\{X \in (x - \epsilon, x + \epsilon]\} = \int_{x-\epsilon}^{x+\epsilon} p(\xi)\,d\xi \approx 2\epsilon\,p(x) \] \[ p(x) = \frac{\Pr\{X \in (x - \epsilon, x + \epsilon]\}}{2\epsilon} \]

Ratio of densities: if \(p(x_1) \gt p(x_2)\), \(X\) is more likely to be close to \(x_1\) than to \(x_2\).

Empirical cdf of Iris petal length with F(5) = 0.72 marked
L2 s8: empirical cdf of Iris petal length. \(F(5) = 0.72\) means 72% of flowers have petal length 5 cm or less. The flat part between 2 and 3 cm is the gap between Setosa and the other species.
\(p(x)\), \(f(x)\)
the density (HW1 writes \(f\))
\(F(x)\)
the cdf, area under the density up to \(x\)
\(\epsilon\)
a tiny half-width around \(x\)

Recipe: probability of an interval from a density

  1. If the density has an unknown constant, set the total area to 1 and solve for it.
  2. Integrate once to get \(F(x)\) (add the "0 below, 1 above" pieces).
  3. \[ \Pr(a \le X \le b) = F(b) - F(a) \]

For a continuous variable \(\le\) and \(\lt\) give the same answer, because a single point has probability 0. For a discrete variable they do not.

Find the constant, the cdf and an interval probability

\(f(x) = x^2 + \frac23 x + \frac13\) for \(0 \le x \le c\). (a) Find \(c\). (b) Find \(F(x)\). (c) Compute \(\Pr(\frac13 \le X \le \frac23)\).

  1. Total area must be 1: \[ \int_0^c \Big(x^2 + \tfrac23 x + \tfrac13\Big)dx = \frac{c^3}{3} + \frac{c^2}{3} + \frac{c}{3} \]
  2. \[ \frac{c^3 + c^2 + c}{3} = 1 \implies c^3 + c^2 + c - 3 = 0 \]
  3. Try \(c = 1\): \(1 + 1 + 1 - 3 = 0\) ✓. So \(c = 1\). (The cubic has no other positive root: its derivative \(3c^2 + 2c + 1\) is always positive, so it is increasing.)
  4. \[ F(x) = \begin{cases} 0, & x \lt 0 \\ \frac{x^3 + x^2 + x}{3}, & 0 \le x \le 1 \\ 1, & x \gt 1 \end{cases} \]
  5. \[ F\!\left(\tfrac23\right) = \frac{\frac{8}{27} + \frac{4}{9} + \frac{2}{3}}{3} = \frac{38}{81} = 0.4691 \]
  6. \[ F\!\left(\tfrac13\right) = \frac{\frac{1}{27} + \frac{1}{9} + \frac{1}{3}}{3} = \frac{13}{81} = 0.1605 \]
  7. \[ \Pr\!\left(\tfrac13 \le X \le \tfrac23\right) = \frac{38 - 13}{81} = \frac{25}{81} = 0.3086 \]

Empirical cdf and pmf: estimating from the sample

You never know the true distribution of a column; you estimate it from the rows you have. The empirical cdf at \(x\) is simply the fraction of samples that are at most \(x\).

The empirical cumulative distribution function of \(X\):

\[ \hat F(x) = \frac1n \sum_{i=1}^n I(x_i \le x) \]

where \(I(\cdot)\) is a binary indicator function:

\[ I(x_i \le x) = \begin{cases} 1, & \text{if } x_i \le x \\ 0, & \text{if } x_i \gt x \end{cases} \]
  • The empirical cdf at \(x\) is the number of samples smaller than or equal to \(x\), divided by \(n\).
  • It is just an estimate of the true cdf.

The empirical probability mass function:

\[ \hat p(x) = \frac1n \sum_{i=1}^n I(x_i = x) \]

It puts a probability mass of \(1/n\) at each point \(x_i\).

A hat \(\hat{\ }\) always means "estimated from the sample". The same convention gives \(\hat\mu\), \(\hat\sigma^2\) and \(\hat\Sigma\) below and in Lectures 3–4.

Empirical cdf of five values

\(X = (1.4, 1.4, 1.3, 1.5, 1.4)\), the first five Iris petal lengths. Find \(\hat F(1.4)\).

  1. Count values \(\le 1.4\): 1.4, 1.4, 1.3, 1.4, that is 4 of 5.
  2. \[ \hat F(1.4) = \frac45 = 0.8 \]
  3. The empirical pmf puts mass \(\frac15\) at 1.3, \(\frac35\) at 1.4 and \(\frac15\) at 1.5.

Part 4

Centre and spread of one attribute

Mean and variance feed straight into the mean vector and covariance matrix of Lecture 3. Median, range and IQR are the robust alternatives.

Mean and sample mean

The mean is the balance point of the distribution. The sample mean is the ordinary average of the column, and it is your estimate of the true mean.

The mean of a random variable \(X\):

\[ \text{Continuous: } \mu_X = E[X] = \int_{-\infty}^{\infty} x\,p(x)\,dx \] \[ \text{Discrete: } \mu_X = E[X] = \sum_{i=1}^n x_i\,p(x_i) \]

The sample mean:

\[ \hat\mu_X = \frac1n \sum_{i=1}^n x_i \]

It serves as an estimate for \(\mu_X\).

Example: Iris petal length.

\[ \hat\mu(\text{Iris}) = 3.758,\quad \hat\mu(\text{Setosa}) = 1.462 \] \[ \hat\mu(\text{Versicolor}) = 4.26,\quad \hat\mu(\text{Virginica}) = 5.552 \]
\(E[\cdot]\)
expectation: a probability-weighted average
\(\mu_X\)
the true mean (unknown)
\(\hat\mu_X\), \(\bar X\)
the sample mean (computed from data)

The sample mean is the empirical pmf plugged into the discrete formula: each \(x_i\) gets weight \(1/n\). The mean of each column together form the mean vector \(\hat{\boldsymbol\mu}\) of Lecture 3.

Sample mean of a small and a 20-value sample

(i) \(X = (18, 18, 20, 22, 22)\). (ii) \(X\) = (69, 74, 68, 70, 72, 67, 66, 70, 76, 68, 72, 79, 74, 67, 66, 71, 74, 75, 75, 76). Find the sample mean of each.

  1. \[ \text{(i)}\quad \hat\mu_X = \frac{100}{5} = 20 \]
  2. \[ \text{(ii)}\quad \sum x_i = 1429 \]
  3. \[ \hat\mu_X = \frac{1429}{20} = 71.45 \]

Median and why it is robust

The median is the middle value after sorting: half the data sits below it. One wild value can drag the mean a long way, but barely moves the median.

The median of an RV is the value \(\nu\) such that

\[ \Pr(X \le \nu_X) = \Pr(X \ge \nu_X) = 0.5 \]

The median is the second quartile. It is the value for which

\[ F(\nu_X) = 0.5,\quad \text{or}\quad \nu_X = F^{-1}(0.5) \]

It can be estimated from the empirical cdf or its inverse:

\[ \hat F(\hat\nu) = 0.5,\quad \text{or}\quad \hat\nu = \hat F^{-1}(0.5) \]

Steps to compute \(\hat\nu\):

  1. Sort the data in increasing order: \(x_1 \le x_2 \le \dots \le x_n\).
  2. If \(n\) is odd, \(\hat\nu = x_{(n+1)/2}\).
  3. If \(n\) is even, \(\hat\nu = \frac12 (x_{n/2} + x_{n/2+1})\).

Robustness (L2 s15): the median is more robust to outliers than the mean.

\(\nu\), \(\hat\nu\)
median (true, estimated)
\(F^{-1}\)
inverse cdf, the quantile function: the \(x\) where the cdf reaches a given level

Sort first. The most common mistake is taking the middle of the unsorted list.

Even and odd samples, and one outlier
  1. \(X = \{1, 2, 1, 5, 4, 3\}\), sorted \(\{1, 1, 2, 3, 4, 5\}\), \(n = 6\) even: \[ \hat\nu = \tfrac12(2 + 3) = 2.5 \]
  2. \(X = \{1, 2, 1, 5, 4, 3, 3\}\), sorted \(\{1, 1, 2, 3, 3, 4, 5\}\), \(n = 7\) odd: \[ \hat\nu = x_4 = 3 \]
  3. \(X = \{1.5, 2, 3.5, 2, 5, 4, 3\}\): \[ \mu = \tfrac{21}{7} = 3 \]
  4. \[ \nu = 3 \]
  5. Replace 4 by 32: \(X = \{1.5, 2, 3.5, 2, 5, 32, 3\}\): \[ \mu = \tfrac{49}{7} = 7 \]
  6. \[ \nu = 3 \] The mean jumped; the median did not move.
Median of the two age samples
  1. (i) \((18, 18, 20, 22, 22)\) is already sorted, \(n = 5\): the middle value is 20, so the median is 20.
  2. (ii) Sorted: 66, 66, 67, 67, 68, 68, 69, 70, 70, 71, 72, 72, 74, 74, 74, 75, 75, 76, 76, 79.
  3. \(n = 20\) even, so average the 10th and 11th: \[ \hat\nu = \frac{71 + 72}{2} = 71.5 \]
  4. (The key also reports the mode, the most frequent value: 74 appears three times.)

Range, quartiles and the inter-quartile range

The range is max minus min. The IQR is the width of the middle half of the data, so a single extreme value cannot inflate it.

Dispersion is the extent to which the distribution is stretched or squeezed. Common measures: range, inter-quartile range, standard deviation, mean absolute deviation, median absolute deviation.

\[ R = x_{\max} - x_{\min} \]

Quartiles are the three values that divide the data into 4 equal parts:

\[ q_1 = F^{-1}(0.25) \] \[ q_2 = \nu_X = F^{-1}(0.5) \] \[ q_3 = F^{-1}(0.75) \] \[ q_4 = \max\{X\} \]

The inter-quartile range is the difference between the 75th and 25th percentiles:

\[ IQR = q_3 - q_1 \] \[ IQR = F^{-1}(0.75) - F^{-1}(0.25) \]

It is the 25% trimmed range, aka the midspread or middle 50%, and a more robust measure of scale than the range.

For a normal distribution with standard deviation \(\sigma\):

\[ IQR \approx 1.349\,\sigma \]
Normal density with the middle 50% shaded between the quartiles
L2 s18: for a normal curve the middle 50% sits within about \(\pm 0.67\sigma\), so \(IQR \approx 1.349\sigma\).

Computing the quartiles by hand

Case A, even sample size \(2n\):

  1. Sort the data and split it into the \(n\) smallest and \(n\) largest values.
  2. \(q_1\) = median of the \(n\) smallest.
  3. \(q_3\) = median of the \(n\) largest.
  4. \(IQR = q_3 - q_1\).

Case B, odd sample size \(2n + 1\):

  1. Sort and divide into a lower and an upper group.
  2. Two alternatives: (a) exclude the median from both groups, giving \(q_1^a, q_3^a\); (b) include the median in both groups, giving \(q_1^b, q_3^b\).
  3. Average the two: \[ q_1 = \tfrac12(q_1^a + q_1^b) \]
  4. \[ q_3 = \tfrac12(q_3^a + q_3^b) \]
  5. \(IQR = q_3 - q_1\).

Software (numpy, Excel) uses other interpolation rules and will give slightly different quartiles. In the exam, follow the slide's method and say which case you used.

Even case: \(X = \{1, 0, -1, 2, -2, 1\}\)
  1. Sorted: \(\{-2, -1, 0, 1, 1, 2\}\). Lower \(\{-2, -1, 0\}\), upper \(\{1, 1, 2\}\).
  2. \[ q_1 = \operatorname{median}\{-2, -1, 0\} = -1 \]
  3. \[ q_3 = \operatorname{median}\{1, 1, 2\} = 1 \]
  4. \[ IQR = 1 - (-1) = 2 \]
Odd case: \(X = \{1, 0, -1, 2, 2, -2, 1\}\)
  1. Sorted: \(\{-2, -1, 0, \mathbf 1, 1, 2, 2\}\), median \(= 1\).
  2. (a) Exclude the median: lower \(\{-2, -1, 0\}\), upper \(\{1, 2, 2\}\). \[ q_1^a = -1 \]
  3. \[ q_3^a = 2 \]
  4. (b) Include the median: lower \(\{-2, -1, 0, 1\}\), upper \(\{1, 1, 2, 2\}\). \[ q_1^b = -0.5 \]
  5. \[ q_3^b = 1.5 \]
  6. \[ q_1 = \tfrac12(-1 - 0.5) = -0.75 \]
  7. \[ q_3 = \tfrac12(2 + 1.5) = 1.75 \]
  8. \[ IQR = 1.75 - (-0.75) = 2.5 \]
All the centre and spread measures for one sample

\(X = \{1, 1, 1.5, 2, 2.5, 3, 4, 5, 7\}\), \(n = 9\).

  1. \[ \mu = \frac{27}{9} = 3 \]
  2. Median: already sorted, odd \(n\), the 5th value: \[ \nu = x_5 = 2.5 \]
  3. Variance (divide by \(n\)): \[ \sigma^2 = \frac{(-2)^2 \times 2 + (-1.5)^2 + (-1)^2 + (-0.5)^2 + 0 + 1^2 + 2^2 + 4^2}{9} \]
  4. \[ \sigma^2 = \frac{32.5}{9} = 3.6111 \]
  5. Quartiles, odd case. Excluding the median: lower \(\{1, 1, 1.5, 2\}\) gives \(q_1^a = 1.25\), upper \(\{3, 4, 5, 7\}\) gives \(q_3^a = 4.5\). Including it: lower \(\{1, 1, 1.5, 2, 2.5\}\) gives \(q_1^b = 1.5\), upper \(\{2.5, 3, 4, 5, 7\}\) gives \(q_3^b = 4\).
  6. \[ q_1 = \frac{1.25 + 1.5}{2} = 1.375 \]
  7. \[ q_3 = \frac{4.5 + 4}{2} = 4.25 \]
  8. \[ R = 7 - 1 = 6 \]
  9. \[ IQR = 4.25 - 1.375 = 2.875 \]
  10. Outlier fences at \(1.5 \times IQR\) beyond the quartiles: \[ 1.375 - 1.5 \times 2.875 = -2.9375 \]
  11. \[ 4.25 + 1.5 \times 2.875 = 8.5625 \] Both 1 and 7 are inside, so there are no outliers.
Repeat with the last value 7 replaced by 16

\(X = \{1, 1, 1.5, 2, 2.5, 3, 4, 5, 16\}\). Find the mean, median, variance, quartiles, range, IQR, and check for outliers.

  1. \[ \mu = \frac{36}{9} = 4 \] (was 3: the mean moved.)
  2. \[ \nu = x_5 = 2.5 \] (unchanged.)
  3. \[ \sigma^2 = \frac{176.5}{9} = 19.61 \] (was 3.61: the variance exploded.)
  4. Quartiles: the lower groups are unchanged, so \(q_1 = 1.375\). Upper groups \(\{3, 4, 5, 16\}\) and \(\{2.5, 3, 4, 5, 16\}\) still have medians 4.5 and 4, so \(q_3 = 4.25\).
  5. \[ R = 16 - 1 = 15 \] (range tripled.)
  6. \[ IQR = 2.875 \] (unchanged.)
  7. Upper fence \(4.25 + 1.5 \times 2.875 = 8.5625\): the value 16 is above it, so it is an outlier. Median and IQR are robust; mean, variance and range are not.

Variance, standard deviation and the sample variance

Variance is the average squared distance from the mean, so it measures spread. Its square root, the standard deviation, is in the same units as the data.

The variance of a random variable \(X\):

\[ \text{Continuous: } \sigma_X^2 = E[(X - \mu_X)^2] = \int_{-\infty}^{\infty} (x - \mu_X)^2 p(x)\,dx \] \[ \text{Discrete: } \sigma_X^2 = E[(X - \mu_X)^2] = \sum_{i=1}^n (x_i - \mu_X)^2 p(x_i) \]

The variable \(\sigma_X\) is the standard deviation. It can easily be shown that

\[ \sigma_X^2 = E[X^2] - \mu_X^2 \]

where \(E[X^2]\) is the second moment.

The maximum likelihood estimator of variance (sample variance):

\[ \hat\sigma^2 = \frac1n \sum_{i=1}^n (x_i - \hat\mu)^2 \]

Geometric interpretation (L2 s23). Write the sample as a vector \(\boldsymbol x = (x_1, \dots, x_n)\) and centre it, \(\tilde{\boldsymbol x} = \boldsymbol x - \hat\mu\). Then

\[ \hat\sigma^2 = \frac1n \tilde{\boldsymbol x}^T \tilde{\boldsymbol x} = \frac1n \|\tilde{\boldsymbol x}\|^2 \]

The sample variance is the squared Euclidean norm of the centred vector, divided by the sample size. It is a random variable and a biased estimate of the true variance:

\[ E[\hat\sigma^2] = \frac{n-1}{n}\,\sigma^2 \longrightarrow \sigma^2 \text{ as } n \to \infty \]

An unbiased estimate is obtained by normalising by \(n - 1\).

\(\sigma_X^2\), \(\sigma_X\)
true variance and standard deviation
\(\hat\sigma^2\), \(s^2\)
sample variance (divide by \(n\)) and the unbiased version (divide by \(n-1\))
\(\tilde{\boldsymbol x}\)
the centred column: every value minus the mean

Recipe: sample variance by hand

  1. Compute the mean.
  2. Subtract it from every value (the centred vector).
  3. Square each, add up.
  4. Divide by \(n\) (or \(n-1\) if the question says so). Square root for the standard deviation.

Lectures and HW2 use \(\frac1n\) (the ML estimator). Your practice sheet notes Q3 and the HW1 key for Q3(c) effectively use \(n-1\). Use what the question says; if it says nothing, use \(\frac1n\) and state it.

Variance is a norm, and covariance (Lecture 3) is a dot product of two centred columns. All of Part 2 applies to data columns, not just to points. The variance of every column together gives the diagonal of the covariance matrix.

Sample variance of a weight column

(i) \(Y = (46, 50, 54, 58, 62)\), use \(n-1\). (ii) \(Y\) = (153, 175, 155, 135, 172, 150, 115, 137, 200, 130, 140, 265, 185, 112, 140, 150, 165, 185, 210, 220), use \(n\).

  1. \[ \text{(i)}\quad \bar Y = \frac{270}{5} = 54 \]
  2. Deviations \((-8, -4, 0, 4, 8)\), squares sum to 160: \[ s_Y^2 = \frac{160}{4} = 40 \]
  3. \[ \text{(ii)}\quad \hat\mu_Y = \frac{3294}{20} = 164.70 \]
  4. \[ \sum (y_i - 164.7)^2 = 27384.2 \]
  5. \[ \hat\sigma_Y^2 = \frac{27384.2}{20} = 1369.21 \]
  6. \[ \hat\sigma_Y = \sqrt{1369.21} = 37.00 \] With \(n-1\): 1441.27.
Use the sample mean and standard deviation in a normal model

Age \(X\) (the 20 values above) has \(\hat\mu_X = 71.45\). Assuming \(X\) is normal with the sample mean and sample standard deviation, what is the probability of an age of 80 or higher? (The normal cdf \(\Phi\) is on the z-score card.)

  1. \[ \sum (x_i - 71.45)^2 = 276.95 \]
  2. \[ \hat\sigma_X^2 = \frac{276.95}{20} = 13.8475 \]
  3. \[ \hat\sigma_X = \sqrt{13.8475} = 3.7212 \]
  4. \[ z = \frac{80 - 71.45}{3.7212} = 2.30 \]
  5. \[ \Pr(X \ge 80) = 1 - \Phi(2.30) = 1 - 0.9893 = 0.0107 \]
  6. With the \(n-1\) standard deviation (3.818): \(z = 2.24\), \(\Pr = 0.0126\). Either is about 1%.

The posted HW1 key writes \(\Pr(X \ge 80) = 0\). That is the empirical answer (no sampled age reaches 80), not the normal-model answer the question asks for. Under the normal model the probability is about 0.011. Show the z-score and the table lookup; that is what earns the marks.

Part 5

Exam toolkit

Which formula, when

The formulas are given in the exam. The skill being tested is picking the right one and showing each step.

The question asks for…UseCard
length / magnitude of a vector\(\|\mathbf x\| = \sqrt{\mathbf x^T\mathbf x}\)Norm
a unit vector\(\mathbf x / \|\mathbf x\|\)Norm
\(L_1\) or \(L_\infty\) normsum of \(|x_i|\), or the largest \(|x_i|\)p-norms
dot product\(\sum x_i y_i\)Inner product
cosine similarity / angle\(\cos\theta = \mathbf x^T\mathbf y / (\|\mathbf x\|\|\mathbf y\|)\)Inner product
are they orthogonal?check \(\mathbf x^T\mathbf y = 0\)Orthogonality
projection of \(\mathbf y\) onto \(\mathbf x\)\(\frac{\mathbf x^T\mathbf y}{\mathbf x^T\mathbf x}\,\mathbf x\)Projection
the part left over after projecting\(\mathbf y - \mathbf y_\parallel\)Projection
distance between two points\(\|\mathbf x_1 - \mathbf x_2\|\) with the stated normDistance
are the columns independent? true dimension?look for one column as a combination of the othersBasis
is this a valid pmf?each \(0 \le p \le 1\), sum \(= 1\)pmf
cdf of a discrete variablerunning sum of the pmfcdf
probability of an interval\(F(b) - F(a)\)cdf pdf
the constant in a densityset \(\int f = 1\)pdf
cdf from a density\(F(x) = \int_{-\infty}^x f\)pdf
fraction of samples \(\le x\)\(\hat F(x) = \frac1n \sum I(x_i \le x)\)Empirical
mean of a sample\(\frac1n \sum x_i\)Mean
mean of a distribution\(\sum x\,p(x)\) or \(\int x\,p(x)\,dx\)Mean
mediansort, take the middle (average the two middles if \(n\) is even)Median
quartiles, IQRmedians of the lower and upper halvesIQR by hand
variance, standard deviation\(\frac1n\sum(x_i - \hat\mu)^2\), then square rootVariance
a probability under a normal model\(z = (x - \hat\mu)/\hat\sigma\), then the \(\Phi\) tableVariance (then Lectures 3–4, z-score)