2.3 Method of Least Squares

Recall the single linear regression model \[Y_i=\alpha +\beta X_i+\epsilon _i, \quad i=1,2,3,\cdots \] We find the best fit by minimizing \(L(\alpha ,\beta )=\sum ^n_{i=1}\left (Y_i-\alpha -\beta X_i\right )^2\). To do this we solve \(\frac {\partial }{\partial \alpha }L(\alpha ,\beta )=0\) and \(\frac {\partial }{\partial \beta }L(\alpha ,\beta )=0\). \[\frac {\partial }{\partial \alpha }L(\alpha ,\beta ) = \sum ^n_{i=1}2(Y_i-\alpha -\beta X_i)\cdot (-1)=0\] \[\sum ^n_{i=1}(Y_i-\alpha -\beta X_i) = 0\] \[n\overline {Y}-n\alpha -n\beta \overline {X}=0\] \begin {equation} \implies \alpha = \overline {Y}-\beta \overline {X} \end {equation} Also \[\frac {\partial }{\partial \beta }L(\alpha ,\beta ) = \sum ^n_{i=1}2(Y_i-\alpha -\beta X_i)\cdot (-X_i)=0\] \begin {equation} \implies \sum ^n_{i=1}Y_iX_i-\alpha \sum ^n_{i=1}X_i-\beta \sum ^n_{i=1}X_i^2=0 \end {equation}

Substituting \(\alpha \) into (2.4) \[\sum ^n_{i=1}Y_iX_i-\left (\overline {Y}-\beta \overline {X}\right )n\overline {X}-\beta \sum ^n_{i = 1} X^2_i=0\] \[\sum ^n_{i = 1} Y_iX_i-n\overline {Y}\,\overline {X}=\beta \sum ^n_{i = 1} X_i^2-\beta n\overline {X}^2\]

\[\implies \hspace {0.5cm} \beta = \frac {\sum ^n_{i = 1} Y_iX_i-n\overline {Y}\overline {X}}{\sum ^n_{i = 1} X^2_i-n\overline {X}^2}\] Therefore \[\hat {\beta }_{LSE}=\frac {\sum ^n_{i = 1} Y_iX_i-n\overline {Y}\,\overline {X}}{X^2_i-n\overline {X}^2}=\frac {\sum ^n_{i = 1}(Y_i-\overline {Y})(X_i-\overline {X})}{\sum ^n_{i = 1}(X_i-\overline {X})^2}\]

\[\hat {\alpha }_{LSE}=\overline {Y}-\overline {X}\left (\frac {\sum ^n_{i = 1} Y_iX_i-n\overline {Y}\, \overline {X}}{\sum ^n_{i = 1} X^2_i-n\overline {X}^2}\right ).\]

Other models

1.: In the cell means model, each \(k\) treatment group has its own distinct mean \(\mu _j\). \[Y_{ij}=\mu _{j}+\varepsilon _{ij},\quad i=1,2,\cdots ,n_j,\quad j=1,2,\cdots ,k\]

Solution. You find \(\mu _j's\) that minimizes \[L\left (\mu _1,\cdots ,\mu _k\right )=\sum ^n_{j=1}\sum ^{n_j}_{i=1}(Y_{ij}-\mu _{j})^2\]
\[\frac {\partial }{\partial \mu _j}L(\mu _1,\mu _2,\cdots ,\mu _k)=\sum ^{n_j}_{i=1}2(Y_{ij}-\mu _{j})(-1).\] Equating to zero we get \[\sum ^{n_j}_{i=1}(Y_{ij}-\mu _j)=0 \implies \sum ^{n_j}_{i=1}Y_{ij}-\sum ^{n_j}_{i=1}\mu _j=0\] \[Y_j-n_j\mu _j=0\] \[\implies \quad n_j\overline {Y}._{.j}-n_j\mu _j=0\] Therefore, \(\hat {\mu }_j=\overline {Y}._{j}\), where \[\overline {Y}._{j}=\frac {1}{n_j}{\sum ^{n_j}_{i=1}Y_{ij}}\quad j=1,2,\cdots ,k.\] □
2.: In the One-Way ANOVA, we model the response \(Y_{ij}\) as a function of an overall mean \(\mu \) and a specific treatment effect \(\tau _j.\) \[Y_{ij}=\mu + \tau _j+\varepsilon _{ij}\, , \quad i=1,2,\cdots ,n_j\, ,\quad j=1,2,\cdots ,k\, \quad \sum ^k_{j=1}\tau =0.\]

Solution. You find \(\mu \) and \(\tau _j's\) that minimizes \[L(\mu ,\tau ,\tau _1,\tau _2,\cdots ,\tau _k)=\sum ^k_{j=1}\sum ^{n_j}_{i=1}(Y_{ij}-\mu -\tau _j)^2.\] Estimating the grand mean \(\mu \): \[\frac {\partial }{\partial \mu }L(\mu ,\tau ,\tau _1,\cdots ,\tau _k)=\sum ^k_{j=1}\sum ^{n_j}_{i=1}2(Y_{ij}-\mu -\tau _j)(-1).\] Equating it to zero we get \[\sum ^k_{j=1}\sum ^{n_j}_{i=1}(Y_{ij}-\mu -\tau _j)=0\] \[\sum ^k_{j=1}\sum ^{n_j}_{i=1}Y_{ij}-kn_j\mu -n_j\sum ^k_{j=1}\tau _j=0\] \begin {equation} \hat {\mu }=\frac {1}{k}\sum ^k_{j=1}\frac {1}{n_j}\sum ^{n_j}_{i=1}Y_{ij}=\overline {Y}.. \end {equation} Estimating treatment effects \((\tau _j)\): \[\frac {\partial L}{\partial \tau _j} = -2\sum ^{n_j}_{i = 1} (Y_{ij} - \mu - \tau _j)\]
equating to zero we get \[\sum ^{n_j}_{i=1}\big (Y_{ij}-\mu -\tau _j\big )=0\]
\[\sum ^{n_j}_{i=1}Y_{ij}-n_j\mu -n_j\tau _j=0\]
\[n_j\overline {Y}._{j}-n_j\mu -n_j\tau _j=0\] \[\implies \quad \hat {\tau }_j=\overline {Y}._{j}-\mu =\overline {Y}._{j}-\overline {Y}..\] □

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.