5.4 Linear Regression

When \(X\) and \(Y\) are linearly related, we can draw a scatter diagram. We can show this relationship between \(X\) and \(Y\). Our interest is draw a straight line which approximates the points.

\[Y_i=\alpha +\beta X_i+e_i,\hspace {1cm}\text {and}\hspace {1cm} e_i=Y_i-\alpha -\beta X_i\] where \(\alpha +\beta X_i\) represents the linear relationship between \(Y\) and \(X_i\) and \(e_i\) are a random errors. \begin {align*} \beta &=\hspace {0.3cm}\text {coefficient of regression of}\hspace {0.2cm}Y\hspace {0.2cm}\text {and}\hspace {0.2cm}X\\ X&=\hspace {0.3cm}\text {independent variable}\\ Y&=\hspace {0.3cm}\text {dependent variable}\\ \alpha &=\hspace {0.3cm}\text {constant} \end {align*}

We assume that there’s independent of \(X_i\) and normally distributed with mean 0 and standard deviation \(\sigma _{Y/X}\).

Our interest is to draw this line by estimating \(\alpha \) and \(\beta \).

5.4.1 The Method of Least Squares

The parameters \(\alpha \) and \(\beta \) are estimated by the method of least squares so called because estimates \(\hat {a}\) and \(\hat {b}\) of \(\alpha \) and \(\beta \) respectively are chosen so as to minimize \(\sum \limits _{i=1}^{n}e_i^2\), the sum of the squared errors (\(n\) is the number of points on the scatter diagram). \begin {align*} Y_i &=\alpha +\beta X_i+e_i\\e_i&=Y_i-\alpha -\beta X_i\\ \sum ^n_{i=1} e^2_i &=\sum ^n_{i=1}(Y_i-\alpha -\beta X_i)^2 \end {align*}

This expression can be varied by varying \(\alpha \) and \(\beta \), \(\hat {a}\) and \(\hat {b}\) estimates of \(\alpha \) and \(\beta \). We choose these \(\alpha \) and \(\beta \) so that \(\sum e^2\) are minimized.

The appropriate values of \(\hat {a}\) and \(\hat {b}\) are found by partial derivatives. \[Y'=0\hspace {0.5cm}\text {stationary point}\] \[\sum ^n_ie_i^2=\sum ^n_i(Y-\alpha -\beta X)^2\hspace {0.2cm}\text {estimate}\hspace {0.2cm}\alpha \hspace {0.2cm}\text {and}\hspace {0.2cm}\beta \hspace {0.2cm}\text {to become}\hspace {0.2cm}\sum ^n_ie^2_i=\sum ^n_i(Y-\hat {a}-\hat {b}X)^2\] We now take partial derivatives with respect to \(\hat {a}\) and \(\hat {b}\).

\begin {align*} \frac {\partial }{\partial \hat {a}}\Bigg (\sum ^n_i(Y-\hat {a}-\hat {b}X)^2\Bigg )&=0\\\\ 2\Bigg (\sum ^n_i(Y-\hat {a}-\hat {b}X)\Bigg )(-1) &=0\\\\ -2\sum ^n_i(Y-\hat {a}-\hat {b}X)&=0 \tag {1}\label {eq:ls1}\\ \end {align*}

\begin {align*} \frac {\partial }{\partial \hat {b}}\Bigg (\sum ^n_i(Y-\hat {a}-\hat {b}X)^2\Bigg )&=0\\\\ -2\sum ^n_iX(Y-\hat {a}-\hat {b}X)&=0 \tag {2}\label {eq:ls2}\\ \end {align*}

\begin {align*} \text {Now using equation (1) we have the following}\\ \sum ^n_i(Y-\hat {a}-\hat {b}X)&=0\\\\ \sum ^n_iY-n\hat {a}-\hat {b}\sum ^n_iX&=0 \tag {3}\label {eq:ls3}\\\\ n\hat {a}=\sum ^n_iY-\hat {b}\sum ^n_iX\hspace {0.5cm}&\implies \hspace {0.5cm} \hat {a}=\frac {1}{n}\Bigg [\sum ^n_iY-\hat {b}\sum ^n_iX\Bigg ]\\\\\\ \implies \hspace {0.5cm}\hat {a}&=\overline {Y}-\hat {b}\overline {X}\\\\ \end {align*}

\begin {align*} \text {Also from } \eqref {eq:ls3} \text {, it follows that}\\ \sum ^n_1X(Y-\hat {a}-\hat {b}X)&=0\\\\ \sum ^n_iXY-\hat {a}\sum ^n_iX-\hat {b}\sum ^n_iX^2&=0\\\\ \sum ^n_iXY-\sum ^n_iX\Bigg [\frac {1}{n}\Bigg (\sum ^n_iY-\hat {b}\sum ^n_iX\Bigg )\Bigg ]-\hat {b}\sum ^n_iX^2&=0\\\\ \sum ^n_iXY-\frac {1}{n}\sum ^n_iX\sum ^n_iY+\frac {1}{n}\hat {b}\Bigg (\sum ^n_iX\Bigg )^2-\hat {b}\sum ^n_iX^2&=0\\ \end {align*}

\begin {align*} \hat {b}\Bigg [\frac {1}{n}\Bigg (\sum ^n_iX\Bigg )^2-\sum ^n_iX^2\Bigg ]&=\frac {1}{n}\sum ^n_iX\sum ^n_iY-\sum ^n_iXY\\\\ \hat {b}&=\frac {\frac {1}{n}\sum \limits ^n_iX\sum \limits ^n_iY-\sum \limits ^n_iXY}{\frac {1}{n}\Bigg (\sum \limits ^n_iX\Bigg )^2-\sum \limits ^n_iX^2}\\\\ \hat {b}&=\frac {\sum X\sum Y-n\sum XY}{\Bigg (\sum X\Bigg )^2-n\sum X^2}\\\\ \implies \hspace {0.5cm} \hat {b}=\frac {n\sum XY-\sum X\sum Y}{n\sum X^2-\Bigg (\sum X\Bigg )^2}\\ \end {align*}

Therefore, the relationship between \(X\) and \(Y\) is given by \(Y=\hat {a}+\hat {b}X\hspace {0.1cm}\) and this is called the Regression line of \(Y\) on \(X\).

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.