3.2 Review of Linear Algebra
Three ideas are collected here because the estimation theory of Section 3.3 cannot be stated without them: the column space of \(X\), which is the set of fitted values the model can produce; projection onto that space, which is what least squares does; and rank, which counts how many parameters the data can distinguish.
The geometric reading is worth carrying forward. Least squares chooses the point of the column space closest to \(\underline {Y}\), the residual vector is perpendicular to that space, and Pythagoras in \(n\) dimensions is the identity \(SST=SSR+SSE\).
- i).
- A vector is an ordered \(n-\) tuple, \(\quad X= \begin {pmatrix} x_1 & x_2 & \cdots & x_n\\ \end {pmatrix}^t \)
- ii).
- Sum of two vectors \(X\) and \(Y\) is defined \(\quad X+Y= \begin {pmatrix} x_1+y_1 & x_2+y_2 & \cdots & x_n+y_n\\ \end {pmatrix}^t \)
- iii).
- Product of a vector and a scalar \(\lambda \), \(\quad \lambda X= \begin {pmatrix} \lambda x_1 & \lambda x_2 & \cdots & \lambda x_n\\ \end {pmatrix}^t \)
- iv).
- \(X\) is a linear combination of vectors \(u_1,u_2,\ldots ,u_{\alpha }\) with coefficients \(\lambda _1,\lambda _2,\ldots ,\lambda _{\alpha }\) if \[X=\lambda _1 u_1 + \lambda _2u_2+\cdots + \lambda _{\alpha }u_{\alpha }\]
- v).
- Inner product (scalar product) of two vectors \(X\) and \(Y\) which is denoted as \(X^tY\) or \(Y^tX\) which is given
by
\[X^tY=Y^tX=\sum ^n_{i=1}x_iy_j\]
which is a real number.
\(X^tX=\sum \limits ^n_{i=1}x^2_i\) is denoted as \(||x||^2.\)
- vi).
- The norm (length) of the vector \(X\) denoted as \(||X||\) and is given by
\[||X||=\sqrt {X^tX}\]
\(X^tX=0\iff X\) is a zero vector.
- i.
- \(X^t\big (Y+Z\big )=X^tY+X^tZ\)
- ii.
- \(X^t\big (\lambda Y\big ) = \lambda X^tY\)
- iii.
- \(X^tX\geq 0\)
- iv.
- \(||\lambda X||=|\lambda |||X||\)
- v.
- \(\big |X^tY\big |^2\leq ||X||^2||Y||^2\) Schwartz-Inequality.
- i.
- Two vectors \(X\) and \(Y\) are linearly independent if and only if \(\quad X^tY=Y^tX=0\)
- ii.
- A basis for a vector space \(V\) is a set of linearly independent vectors that span \(V\). \[Z_1= \begin {pmatrix} 1\\ 0\\ \end {pmatrix} ,\quad Z_2= \begin {pmatrix} 0\\1\ \end {pmatrix} \]
Lemma 3.2.4. Let vector space \(V_{\alpha }\) be a subspace of \(V_n\) i.e \(V_{\alpha }\subset V_n\) and \(X\in V_n\). There exists vectors \(Y\) and \(Z\) such that \(X=Y+Z,\quad Y\in V_{\alpha }\) and \(Z\perp V_{\alpha }\). (i.e \(Z\) is orthogonal to \(V_{\alpha }\)). This decomposition is unique.
Definition 3.2.5. Given a vector \(X\in V_n\), the vector \(Y\in V_r\subset V_n\) given in Lemma 3.2.4 above is such that \(X-Y\perp V_r\), then \(Y\) is called a projection of \(X\) on \(V_r\).
Theorem 3.2.6. Given a fixed vector space \(V_r\subset V_n\), a fixed vector \(X\in V_n\) and a variable vector \(Y\in V_r\), then \(||X-Y||\) has a minimum value if and only if \(Y\) is a projection of \(X\) on \(V_r\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.