2.3 Method of Least Squares
In the simple linear regression model \[Y_i = \alpha + \beta x_i + \epsilon _i ,\qquad i = 1,2,\dots ,n,\] the least squares estimators of \(\alpha \) and \(\beta \) minimise the sum of squared deviations \[L(\alpha ,\beta ) = \sum ^n_{i=1}\left (Y_i - \alpha - \beta x_i\right )^2 .\]
Theorem 2.3.1. The least squares estimators in simple linear regression are \[\hat {\beta } = \frac {\sum ^n_{i = 1}(x_i - \bar {x})(Y_i - \bar {Y})}{\sum ^n_{i = 1}(x_i - \bar {x})^2} = \frac {\sum ^n_{i = 1} x_iY_i - n\bar {x}\bar {Y}}{\sum ^n_{i = 1} x^2_i - n\bar {x}^2},\qquad \hat \alpha = \bar Y - \hat \beta \bar x .\]
Proof. Setting the partial derivatives to zero, \[\frac {\partial L}{\partial \alpha } = -2\sum ^n_{i=1}(Y_i - \alpha - \beta x_i) = 0,\qquad \frac {\partial L}{\partial \beta } = -2\sum ^n_{i=1}x_i(Y_i - \alpha - \beta x_i) = 0 .\] The first gives \(n\bar {Y} - n\alpha - n\beta \bar {x} = 0\), so \(\alpha = \bar {Y} - \beta \bar {x}\). Substituting into the second, \[\sum ^n_{i=1}x_iY_i - \left (\bar {Y} - \beta \bar {x}\right )n\bar {x} - \beta \sum ^n_{i = 1} x^2_i = 0,\qquad \text {so}\qquad \beta \left (\sum x_i^2 - n\bar x^2\right ) = \sum x_iY_i - n\bar x\bar Y ,\] which gives \(\hat \beta \); the two forms agree because \(\sum (x_i - \bar x)(Y_i - \bar Y) = \sum x_iY_i - n\bar x\bar Y\) and \(\sum (x_i - \bar x)^2 = \sum x_i^2 - n\bar x^2\). Since \(L\) is a convex quadratic, the stationary point is the minimum. □
The same principle estimates other linear models.
- 1.
- In the cell means model, each of \(k\) treatment groups has its own mean: \[Y_{ij} = \mu _{j} + \varepsilon _{ij},\qquad i = 1,\dots ,n_j,\quad j = 1,\dots ,k .\] Minimising \(L = \sum _{j=1}^k\sum _{i=1}^{n_j}(Y_{ij} - \mu _{j})^2\), the equation \(\frac {\partial L}{\partial \mu _j} = -2\sum _{i=1}^{n_j}(Y_{ij} - \mu _j) = 0\) gives \(\hat {\mu }_j = \bar {Y}_{\cdot j} = \frac {1}{n_j}\sum _{i=1}^{n_j}Y_{ij}\), the group mean.
- 2.
- In the one-way analysis of variance model, the response is an overall mean plus a treatment effect, \[Y_{ij} = \mu + \tau _j + \varepsilon _{ij},\qquad \sum ^k_{j=1}n_j\tau _j = 0 .\] The fitted values \(\hat \mu + \hat \tau _j\) must equal the group means \(\bar Y_{\cdot j}\), as in the cell means model, and the constraint then gives \(\hat \mu = \frac {1}{N}\sum _j n_j\bar Y_{\cdot j} = \bar Y_{\cdot \cdot }\), the grand mean of all \(N = \sum n_j\) observations, and \(\hat \tau _j = \bar {Y}_{\cdot j} - \bar {Y}_{\cdot \cdot }\).
Example 2.3.2. Fit the least squares line to the points \((1,2)\), \((2,3)\), \((3,5)\), \((4,4)\), \((5,6)\).
Solution. Here \(\bar x = 3\), \(\bar y = 4\), \(\sum (x_i - \bar x)(y_i - \bar y) = (-2)(-2) + (-1)(-1) + 0 + (1)(0) + (2)(2) = 9\) and \(\sum (x_i - \bar x)^2 = 10\). So \(\hat \beta = 0.9\) and \(\hat \alpha = 4 - 0.9\cdot 3 = 1.3\): the fitted line is \(\hat y = 1.3 + 0.9x\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.