4.4 Meaning of the Coefficients
The single most misread quantity in applied statistics is a regression coefficient. In simple regression \(\beta _1\) is the change in \(Y\) per unit change in \(X\); in multiple regression \(\beta _j\) is the change in \(Y\) per unit change in \(X_j\) with the other regressors held fixed, and that qualification changes everything.
It means the coefficient depends on which other variables are in the model, so that the same variable may have opposite signs in two defensible models of the same data. It means the coefficient is estimated from whatever variation in \(X_j\) remains after the other regressors are accounted for — little variation left, large standard error, which is the subject of Section 4.5. And it means that “holding fixed” is arithmetic performed on the data, not an intervention performed on the world.
\[E\big (Y/X_1,X_2,\ldots ,X_{k-1}\big ) = \sum ^{k-1}_{j=0}\beta _jX_j\] The parameter \(\beta _0\) is the \(Y\) intercept of the regression plane, it is the mean response at the origin. Though in certain situations \(\beta _0\) has no practical importance. The parameter \(\beta _j\) indicates change in the response variable \(Y\) per unit increase in \(X_j=\big (j=1,2,\ldots ,k-1\big )\) when the other variables are held constant. The regression model is a linear model, with design matrix having its columns as vectors of the independent variables \[Y=XB+\varepsilon \quad ,\quad E(\varepsilon )=0\] \begin {align*} Cov(\varepsilon ) & = \sigma ^2I_n,\quad \text {there}\quad E(Y)=XB\\ Cov(Y) & = \sigma ^2I\\ \end {align*}
\begin {align*} \big (X^tX\big )\widehat {B} & = X^tY\\ \implies \quad \widehat {B} & = \big (X^tX\big )^{-1}X^tY\\ \widehat {Y}= X\widehat {B} & = X\big (X^tX\big )^{-1}X^tY=P_vY \end {align*}
where \(P_v=X\big (X^tX\big )^{-1}X^t\) is the projection matrix. we denote \(P_v\) as \(H\) \[\widehat {Y}=HY,\quad H^t=H,\quad \big (I-H\big )^2=I-H\] \[\text {SST}=\sum ^n_{i=1}\big (Y_i-\overline {Y}\big )^2=Y^t\Big (I-\frac {J_n}{n}\Big )Y\] \[\frac {1_n1^t_n}{n}=\frac {J_n}{n}\] \begin {align*} \text {SSE} & = \sum ^n_{i=1}\big (Y-\widehat {Y}\big )^2= \big (Y-\widehat {Y}\big )^t\big (Y-\widehat {Y}\big )\\ & = Y^t\big (I-H\big )Y;\quad H=X\big (X^tX\big )^{-1}X^t\\ & = \quad \text {error sum of squares}\\ & = \quad \text {Residual sum of squares} \end {align*}
\begin {align*} \text {SSR} & = \quad \text {Regression sum of squares}\\ \end {align*}
\begin {align*} \text {SST} & = \text {SSR} + \text {SSE}\\ Y^t\Big (I-\frac {J_n}{n}\Big )Y & = \text {SSR} + Y^t(I-H)Y\\ \text {SSR} & = Y^t\Big (I-\frac {J_n}{n}\Big )Y-Y^t(I-H)Y\\ & = Y^t\Big [I-\frac {J_n}{n}-I+H\Big ]Y \end {align*}
\begin {align*} \text {SSR} & = \sum ^n_{i=1}\big (\widehat {Y}_i-\overline {Y}\big )^2\\ & = Y^t\Big (H-\frac {J_n}{n}\Big )Y\quad \text {Regression sum of squares}\\ Y_i-\overline {Y} & = \widehat {Y}_i-\overline {Y} + \big (Y_i-\widehat {Y}_i\big )\\ \text {SST} & = \text {SSR} + \text {SSE}\\ n-1 & = (k-1) + (n-k) \end {align*}
| Source of | SS | df | MS | F |
| variation | ||||
| Regression | SSR | \(k-1\) | \(\text {MSR}=\frac {\text {SSR}}{k-1}\) | \(\frac {\text {MSR}}{\text {MSE}}\thicksim f_{k-1,n-k}\) |
| under \(H_0\) | ||||
| Error | SSE | \(n-k\) | \(\text {MSE}=\frac {\text {SSE}}{n-k}\) | \(H_0:\beta _j=0\) |
| (residual) | \(j=1,2,\ldots ,k-1\) | |||
| Total | SST | \(n-1\) | ||
\(R^2=\) Coefficient of multiple determination.
\(R^2=\frac {\text {SSR}}{\text {SST}}\quad \) Proportion of variation in \(Y\) about \(\overline {Y}\) explained by the regression model \begin {align*} R^2 & = \frac {\text {SSR}}{\text {SST}}\\ & = 1-\frac {\text {SSE}}{\text {SST}} \end {align*}
\(R^2=1-\) Proportion of variation in \(Y\) about \(\overline {Y}\) unexplained by the regression model.
Alternatively \(R^2\) is the proportionate reduction of the total variation in \(Y\) about \(\overline {Y}\) associated with the set of
variables \(X_1,X_2,\ldots , X_{k-1}\).
\(R^2=0\), if \(\hat {\beta }_j=0,\quad j=1,2,\ldots ,k-1\).
\(R^2=1\), when there is a perfect fit. i.e all the \(Y_i's\) lie on the regression surface.
However, adding more independent (regression) variables to model can only increase \(R^2\) because SSE can
never become larger with more independent variables.
Hence, the need for adjusting \(R^2\) to recognise the number of independent variables in model
\begin {align*} R^2_a & = 1-\frac {\text {SSE}/(n-k)}{\text {SST}/(n-1)}=1-\frac {\text {MSE}}{\text {MST}}\\ & = \,\text {adjusted coefficients of multiple determination}\\ \end {align*}
Full model: \(Y_i=\beta _0+\beta _1X_{i1}+\beta _2X_{i2}+\beta _4X_{i4}+e_i\quad i=1,2,\ldots ,n\)
\[\text {SST} = \text {SSR} + \text {SSE}\quad R^2_4=1-\frac {\text {MSE}}{\text {MST}}\]
\[Y_i=\beta _0+\beta _1X_{i1}+\beta _3X_{i3}+\beta _4X_{i4}+e_i\quad i=1,2,\ldots ,n\]
\[\text {SST} = \text {SSR}(X_2) +\text {SSE}(X_2)\]
\[R^2_a(X_2)=1-\frac {\text {MSE}(X_2)}{\text {MST}}\]
\[R^2_a<R^2_a(X_2)\]
SST \(=\) SSR \(+\) SSE \[Y=\sum _{j=0}X_j\beta _j+e\qquad Y_i=\sum _{j=0}X_{ij}\beta _j+e_i\quad i=1,2,\ldots ,n\]
SST \(=\) Total sum of squares
SSR \(=\) Regression sum of squares. SSE \(=\) Residual (error) sum of squares.
If we have the model \(Y=\beta _0+x_1\beta _1+x_2\beta _2+x_3\beta _3+e\)
SST \(=\) SSR\(\big (X_1,X_2,X_3\big )\) \(+\) SSE\(\big (X_1,X_2,X_3\big )\)
If the model is \(Y=\beta _0+x_2\beta _2+x_3\beta _3+e\) then
SST \(=\) SSR\(\big (X_2,X_3\big )\) + SSE\(\big (X_2,X_3\big )\)
Note.
- SSR\(\big (X_2,X_3\big )<\) SSR\(\big (X_1,X_2,X_3\big )\)
- SSE\(\big (X_2,X_3\big ) >\) SSE\(\big (X_1,X_2,X_3\big )\)
If the model is \(Y=\beta _0+\beta _1X_1+\beta _2X_2+e\)
SST = SSR\(\big (X_1,X_2\big )\) + SSE\(\big (X_1,X_2\big )\)
SSR\(\big (X_1,X_2\big )<\) SSR\(\big (X_1,X_2,X_3\big )\) and SSE\(\big (X_1,X_2\big )>\) SSE\(\big (X_1,X_2,X_3\big )\).
SSE\(\big (X_1/X_2\big )=\) Reduction of SSE when \(X_1\) is introduced given that \(X_2\) is already in model.
SSE\(\big (X_2/X_1\big )=\) SSE\((X_1)-\) SSE\(\big (X_1,X_2\big )\).
SSE\(\big (X_3/X_1,X_2\big )=\) SSE\(\big (X_1,X_2\big )-\) SSE\(\big (X_1,X_2,X_3\big )\)
Reduction in the SSE when \(X_3\) is introduced given that \(X_1\) and \(X_2\) are in model
SSE\(\big (X_2,X_1/X_3)=\) SSE\(\big (X_3\big )-\) SSE\(\big (X_1,X_2,X_3\big )\)
\(X_1\) (first) \(\qquad X_3\)(second) \(\qquad X_2\)(third)
\begin {align*} \text {SSR}\big (X_3/X_1\big ) & = \text {SSE}\big (X_1\big )-\text {SSE}\big (X_1,X_3\big )\\ & = \text {SST}-\text {SSR}\big (X_1\big )-\Big (\text {SST}-\text {SSR}\big (X_1,X_3\big )\Big )\\ \text {SSR}\big (X_3/X_1\big ) & = \text {SSR}\big (X_1,X_3\big )-\text {SSR}\big (X_1\big )\quad \cdots \, (a) \end {align*}
increase of the SSR when \(X_3\) is introduced given that \(X_1\) is already in the model
\[\text {SSR}\big (X_2/X_1,X_3\big )=\text {SSR}\big (X_1,X_2,X_3\big )-\text {SSR}\big (X_1,X_3\big )\quad \cdots \quad (b)\] increase of the SSR when \(X_2\) is introduced given that \(X_1\) and \(X_3\) is already in the model
\begin {align*} \underbrace {\text {SST}}_{n-1} & = \underbrace {\text {SSR}\big (X_1,X_2,X_3\big )}_3+\underbrace {\text {SSE}\big (X_1,X_2,X_3\big )}_{n-4}\\ & = \text {SSR}\big (X_2/X_1,X_3\big )+\text {SSR}\big (X_1,X_3\big )+\text {SSE}\big (X_1,X_2,X_3\big )\quad \text {from}\quad (b)\\ & = \text {SSR}\big (X_2/X_1,X_3\big )+\text {SSR}\big (X_3/X_2\big )+\text {SSR}\big (X_1\big )+\text {SSE}\big (X_1,X_2,X_3\big )\quad \text {from}\quad (a)\\ \text {SST} & = \text {SSR}\big (X_1\big )+\text {SSR}\big (X_3/X_1\big )+\text {SSR}\big (X_2/X_1,X_3\big )+\text {SSE}\big (X_1,X_2,X_3\big ) \end {align*}
| Source of | SS | df | MS | F |
| Variation | ||||
| Regression | SSR\(\big (X_1,X_2,X_3\big )\) | 3 | MSR\(\big (X_1,X_2,X_3\big )\) | \(H_0:\beta _j=0,\quad j=1,2,3\) |
| \(X_1\) | SSR\((X_1)\) | 1 | MSR\((X_1)\) | \(H_0:\beta _1=0\) |
| \(X_3/X_1\) | SSR\((X_3/X_1)\) | 1 | MSR\((X_3/X_1)\) | do we need \(X_3\) given \(X_1\) |
| is already in the model | ||||
| \(X_2/X_1,X_3\) | SSR\((X_2/X_1,X_3\) | 1 | MSR\((X_2/X_1,X_3)\) | do we need \(X_2\) given \(X_1\) and \(X_3\) |
| is already in the model | ||||
| Error | SSE\((X_1,X_2,X_3)\) | \(n-4\) | MSE\((X_1,X_2,X_3)\) | |
| Total | SST | \(n-1\) | ||
First \((X_2)\) \(\quad \), second \((X_1\) and \(X_3)\) \[\text {SSR}\big (X_1,X_2,X_3\big ) = \text {SSR}\big (X_2\big ) + \text {SSR}\big (X_1,X_3/X_2\big )\]
- 1.
- The department of Fisheries (DOF) wants to estimate the number of breams caught in
some lakes in order to restock the lakes with appropriate number of fingerling (body fish).
The DOF samples and record \(Y_1\) the seasonal catch per square Km of lake area, \(X_1\), the number
of lake shore residents per square Km, \(X_2\), the size of the lake in square Km, \(X_3=1\), if the lake has
public access, 0 if not, and \(X_4\), a structure index. Twenty random samples were taken, the
fitted model
\[Y=-2.78+0.0268X_1+0.504X_2+0.743X_3+0.0511X_4\]
- (a)
- Complete the ANOVA TABLE and test for significance of the model
Source SS df MS F Regression 24.0624 4 6.0156 39.655\(\thicksim f_{4,15}^{0.05}\) Error 2.2756 15 0.1517 Total 26.338 19 1.386 - (b)
- Compute the unadjusted coefficients of multiple determination.
- (c)
- The DOF is uncertain whether access \((X_3)\) and structure \((X_4)\) are useful as additional independent
variables. A model was fit on the same data with only \(X_1\) and \(X_2\) and the regression sum of
squares was 12.913. Are \(X_3\) and \(X_4\) useful additional predictors for estimating the number of
breams caught.
Solution.
- (a)
- \begin {align*} R^2 & = \frac {\text {SSR}}{\text {SST}}=1-\frac {\text {SSE}}{\text {SST}}\\ & = \frac {24.0624}{26.338}=1-\frac {2.2656}{26.338}\\ & = 0.9136\\ \\ R^2_a & = 1-\frac {\text {MSE}}{\text {MST}}\\ & = 1-\frac {0.1517}{1.386}\\ & = 0.891\\ \end {align*}
\[ \text {adjusted}\qquad R^2=\frac {\text {SSR}}{\text {SST}}=\frac {12.913}{26.336}\] \[R^2_a=1-\frac {(26.38-12.913)/17}{1.386}=0.43\]
- (b)
- SSR\(\big (X_3,X_4/X_1,X_2\big )=\) SSR\(\big (X_1,X_2,X_3,X_4\big )-\) SSR\((X_1,X_2)\)
Source SS df MS F Regression 24.0624 4 \(X_1,X_2\) 12.913 2 \(X_3,X_4/X_1,X_2\) 11.1494 2 \(\frac {11.1404}{2}\) \(F\thicksim f_{2,15}\) Error 2.2756 15 0.1517 Total 26.338 19 We need the \(X_3\) and \(X_4\).
- 2.
- The heat evolved in Calories per gram of cement\((Y)\) as a function of each of four ingredients in
the mix, aluminate \((X_1)\), silicate \((X_2)\), alumino-ferrite \((X_3)\) and dicalcium \((X_4)\), hence the following
model:
\[Y=\beta _0+\beta _1X_1+\beta _2X_2+\beta _3X_3+\beta _4X_4+e.\]
The fitted model on thirteen batches is
\[\widehat {Y}=62.41+1.55X_1+0.51X_2+0.20X_3-0.14X_4\]
- (a)
- Complete the ANOVA TABLE
Source SS df MS F Regression 2667.90 4 667.0 111.5 Residual 47.86 8 5.983 Total 2715.76 12 226.3 - (b)
- Compute \(R^2\) and \(R^2_a\).
- (c)
- What are the interpretations of the coefficients of \(X_2\) and \(X_4\)?
\(\implies (\) the evolved increases by 0.51 per unit silicate)
- 3.
- For the model in (2) above, the reduced models with their SSR are given as follow
Model SSR \(Y=\beta _0+\beta _4X_4+e\) 1831.90 \(Y=\beta _0+\beta _1X_1+\beta _2X_2+e\) 2641.00 \(Y=\beta _0+\beta _1X_1+\beta _4X_4+e\) 2657.86 \(Y=\beta _0+\beta _1X_1+\beta _2X_2+\beta _4X_4+e\) 2667.79 - (a)
- Compute \(R^2\) and \(R^2_a\) for \[Y=\beta _0+\beta _1X_1+\beta _4X_4+e\]
- (b)
- Decompose the SSR for the model
- (c)
- Test whether \(X_2\) and \(X_3\) are needed in the model which already has \(X_1\) and \(X_4\).
Solution.
- (a)
- The model in \(X_1\) and \(X_4\) has \(\text {SSR}=2657.86\), and \(\text {SST}\) does not depend on which model is fitted, so \(\text {SST}=2715.76\) with \(n-1=12\) degrees of freedom and \(n=13\). Hence \[\text {SSE}=2715.76-2657.86=57.90, \qquad n-k=13-3=10,\] \[R^{2}=\frac {2657.86}{2715.76}=0.9787,\] \[R^{2}_{a}=1-\frac {\text {SSE}/(n-k)}{\text {SST}/(n-1)} =1-\frac {57.90/10}{2715.76/12}=1-\frac {5.790}{226.31}=0.9744.\] The full four-variable model gave \(R^{2}=2667.90/2715.76=0.9824\). Dropping \(X_2\) and \(X_3\) costs 0.0037 of \(R^{2}\) and raises the adjusted value, which is the first sign of what part (c) confirms.
- (b)
- Each line is the extra regression sum of squares brought by a variable given those
above it, so the entries add to the regression sum of squares of the full model.
Source SS d.f. \(X_4\) 1831.90 1 \(X_1\mid X_4\) 825.96 1 \(X_2\mid X_1,X_4\) 9.93 1 \(X_3\mid X_1,X_2,X_4\) 0.11 1 \(X_1,X_2,X_3,X_4\) 2667.90 4 Table 4: Sequential decomposition of the regression sum of squares. The four extra sums of squares total \(1831.90+825.96+9.93+0.11=2667.90\), which is the regression sum of squares of the full model. - (c)
- The two variables are tested together, against the model that already contains \(X_1\) and \(X_4\): \[\text {SSR}\big (X_2,X_3\mid X_1,X_4\big ) =\text {SSR}\big (X_1,X_2,X_3,X_4\big )-\text {SSR}\big (X_1,X_4\big ) =2667.90-2657.86=10.04,\] on \(2\) degrees of freedom. With the full model’s \(\text {MSE}=5.983\), \[F=\frac {\text {SSR}\big (X_2,X_3\mid X_1,X_4\big )/2}{\text {MSE}} =\frac {10.04/2}{5.983}=0.84,\] to be compared with \(F_{2,8,0.05}=4.46\). Since \(0.84<4.46\) the hypothesis \(\beta _2=\beta _3=0\) is not rejected: \(X_2\) and \(X_3\) are not needed once \(X_1\) and \(X_4\) are in the model.
Note. The conclusion is about \(X_2\) and \(X_3\) jointly, and given \(X_1\) and \(X_4\). It does not say either variable is unrelated to \(Y\): fitted alone, \(X_2\) explains a great deal. What the test says is that whatever they carry is already carried by \(X_1\) and \(X_4\). This is why the decomposition of part (b) is written as a sequence — change the order of entry and the individual lines change, though their total does not.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.