2.13 Practice problems

Try each problem before reading the solution.

Problem 2.13.1. The following data were collected:

$x$	2	4	6	8	10

$y$	3	7	8	12	15

(a): Fit the least-squares line.
(b): Compute the fitted values and residuals, and check that $\sum e_i = 0$ and $\sum x_i e_i = 0$.
(c): Estimate $\sigma ^2$ in two ways, and find $\operatorname {se}(\hat \beta _1)$.

Show solution

Solution. (a) $\sum x_i = 30$, $\sum y_i = 45$, $\sum x_i^2 = 220$, $\sum x_i y_i = 328$, $\sum y_i^2 = 491$, so $\bar x = 6$ and $\bar y = 9$. Then \[ S_{xx} = 220 - \frac {30^2}{5} = 40, \qquad S_{xy} = 328 - \frac {30 \times 45}{5} = 58, \] $\hat \beta _1 = 58/40 = 1.45$, $\hat \beta _0 = 9 - 1.45 \times 6 = 0.3$, and $\hat y = 0.3 + 1.45x$.

(b)

$x_i$	2	4	6	8	10

$\hat y_i$	3.2	6.1	9.0	11.9	14.8
$e_i$	$-0.2$	0.9	$-1.0$	0.1	0.2

$\sum e_i = -0.2 + 0.9 - 1.0 + 0.1 + 0.2 = 0$ and $\sum x_i e_i = -0.4 + 3.6 - 6.0 + 0.8 + 2.0 = 0$.

(c) Directly, $SS_{\mathrm {Res}} = 0.04 + 0.81 + 1.00 + 0.01 + 0.04 = 1.90$. By Theorem 2.5.2, $S_{yy} = 491 - 45^2/5 = 86$ and $SS_{\mathrm {Res}} = 86 - 1.45 \times 58 = 86 - 84.1 = 1.90$. So $\hat \sigma ^2 = 1.90/3 = 0.6333$ and $\operatorname {se}(\hat \beta _1) = \sqrt {0.6333/40} = 0.1258$.

Problem 2.13.2. For a sample of $n = 10$ weeks, $x$ is advertising spend (in hundreds of dollars) and $y$ is sales (in thousands of units). Only the totals were kept: \begin {gather*} \sum x_i = 50, \quad \sum y_i = 120, \quad \sum x_i^2 = 310, \\ \sum x_i y_i = 660, \quad \sum y_i^2 = 1550 . \end {gather*} Find the fitted line, estimate the mean sales when $\$600$ is spent, and estimate $\sigma .$

Show solution

Solution. $\bar x = 5$, $\bar y = 12$, $S_{xx} = 310 - 50^2/10 = 60$, $S_{xy} = 660 - 50 \times 120/10 = 60$ and $S_{yy} = 1550 - 120^2/10 = 110$. So $\hat \beta _1 = 60/60 = 1$, $\hat \beta _0 = 12 - 1 \times 5 = 7$, and $\hat y = 7 + x$.

$\$600$ is $x = 6$, so $\hat y = 13$, that is, $13\,000$ units.

$SS_{\mathrm {Res}} = 110 - 1 \times 60 = 50$, $\hat \sigma ^2 = 50/8 = 6.25$, and $\hat \sigma = 2.5$ thousand units. Note that the residuals themselves were never needed: Theorem 2.5.2 works from the totals alone.

Problem 2.13.3. Regression through the origin. Sometimes theory says the line must pass through the origin, giving the model $y_i = \beta _1 x_i + \varepsilon _i$.

(a): Show that the least-squares estimator is $\hat \beta _1 = \sum x_i y_i \big / \sum x_i^2$.
(b): Under A1–A3, show that $\hat \beta _1$ is unbiased with variance $\sigma ^2 \big / \sum x_i^2$.
(c): Fit the model to the data below, and show that here $\sum e_i \neq 0$ but $\sum x_i e_i = 0$.

$x$	1	2	3	4	5	6

$y$	2.1	3.9	6.2	7.8	10.1	11.9

Show solution

Solution. (a) $S(\beta _1) = \sum (y_i - \beta _1 x_i)^2$ has $S'(\beta _1) = -2\sum x_i (y_i - \beta _1 x_i)$. Setting this to zero gives $\sum x_i y_i = \hat \beta _1 \sum x_i^2$. Since $S''(\beta _1) = 2\sum x_i^2 > 0$, this is a minimum.

(b) $\hat \beta _1 = \sum k_i y_i$ with $k_i = x_i / \sum x_j^2$. Then $E(\hat \beta _1) = \sum k_i \beta _1 x_i = \beta _1 \sum x_i^2 / \sum x_j^2 = \beta _1$, and $\operatorname {Var}(\hat \beta _1) = \sigma ^2 \sum k_i^2 = \sigma ^2 \sum x_i^2 / \left (\sum x_j^2\right )^2 = \sigma ^2 / \sum x_i^2$.

(c) $\sum x_i y_i = 2.1 + 7.8 + 18.6 + 31.2 + 50.5 + 71.4 = 181.6$ and $\sum x_i^2 = 91$, so $\hat \beta _1 = 181.6/91 = 1.9956$ and $\hat y = 1.9956x$. Then \begin {align*} \sum e_i &= \sum y_i - \hat \beta _1 \sum x_i = 42 - 1.9956 \times 21 \\ &= 42 - 41.9077 = 0.0923 \neq 0 , \end {align*}

while $\sum x_i e_i = \sum x_i y_i - \hat \beta _1 \sum x_i^2 = 181.6 - 181.6 = 0$. The only normal equation is the one for $\beta _1$, which is $\sum x_i e_i = 0$. The equation that gave $\sum e_i = 0$ belonged to $\beta _0$, and this model has no $\beta _0$.

Problem 2.13.4. For the advertising data, fit the regression of $x$ on $y$, that is, the line $\hat x = \hat \alpha _0 + \hat \alpha _1 y$ that minimises $\sum (x_i - \alpha _0 - \alpha _1 y_i)^2$. Is it the same line as $\hat y = -0.1 + 0.7x$ solved for $x$? Show that in general the product of the two slopes, $\hat \beta _1 \hat \alpha _1$, lies between $0$ and $1$.

Show solution

Solution. By Theorem 2.2.4 with the roles of $x$ and $y$ exchanged, $\hat \alpha _1 = S_{xy}/S_{yy} = 7/6 = 1.1667$ and $\hat \alpha _0 = \bar x - \hat \alpha _1 \bar y = 3 - (7/6)(2) = 0.6667$.

Solving $\hat y = -0.1 + 0.7x$ for $x$ gives $x = (y + 0.1)/0.7$, with slope $1/0.7 = 1.4286$. The two lines are different: one minimises vertical distances, the other horizontal ones.

In general \[ \hat \beta _1 \hat \alpha _1 = \frac {S_{xy}}{S_{xx}} \cdot \frac {S_{xy}}{S_{yy}} = \frac {S_{xy}^2}{S_{xx} S_{yy}} \ge 0 . \] By the Cauchy–Schwarz inequality, $\left ( \sum a_i b_i \right )^2 \le \sum a_i^2 \sum b_i^2$. With $a_i = x_i - \bar x$ and $b_i = y_i - \bar y$ this says $S_{xy}^2 \le S_{xx} S_{yy}$, so the product is at most $1$. It equals $1$ exactly when the points lie on a straight line, and only then do the two regression lines coincide. Here $\hat \beta _1 \hat \alpha _1 = 49/60 = 0.8167$. This quantity is $r^2$, the square of the correlation coefficient of Subsection 2.12.

Problem 2.13.5. Continue with the data of Problem 2.13.1, assuming A1–A4.

(a): Test $H_0\colon \beta _1 = 0$ against $H_1\colon \beta _1 \neq 0$ at the 5% level, using the $t$-test.
(b): Set out the analysis of variance table, carry out the $F$-test, and check that $F_0 = t_0^2$.
(c): Find $R^2$ and the correlation coefficient $r$.
(d): Find a 95% confidence interval for $\beta _1$.
(e): At $x_0 = 7$, find a 95% confidence interval for the mean response and a 95% prediction interval for a new observation.

Show solution

Solution. From Problem 2.13.1: $n = 5$, $\bar x = 6$, $S_{xx} = 40$, $S_{xy} = 58$, $S_{yy} = 86$, $\hat \beta _1 = 1.45$, $\hat \beta _0 = 0.3$, $SS_{\mathrm {Res}} = 1.90$, $\hat \sigma ^2 = 0.6333$ and $\operatorname {se}(\hat \beta _1) = 0.1258$. There are $n - 2 = 3$ degrees of freedom, and $t_{0.025,3} = 3.182$.

(a) $t_0 = 1.45/0.1258 = 11.52$. Since $|t_0| > 3.182$, reject $H_0$: there is a significant linear relationship.

(b) $SS_R = \hat \beta _1 S_{xy} = 1.45 \times 58 = 84.1$.


Source	SS	df	MS	$F_0$

Regression	84.10	1	84.10	132.79
Residual	1.90	3	0.6333

Total	86.00	4

$F_{0.05,1,3} = 10.13$, and $132.79 > 10.13$, so reject $H_0$. Also $t_0^2 = 11.523^2 = 132.79 = F_0$.

(c) $R^2 = 84.1/86 = 0.9779$, so the line explains about $97.8\%$ of the variation in $y$. $r = +\sqrt {0.9779} = 0.9889$, positive because $S_{xy} > 0$.

(d) $1.45 \pm 3.182 \times 0.1258 = 1.45 \pm 0.4004$, that is, $(1.0496,\ 1.8504)$.

(e) $\hat y_0 = 0.3 + 1.45 \times 7 = 10.45$, and $\frac 1n + \frac {(x_0 - \bar x)^2}{S_{xx}} = 0.2 + \frac {1}{40} = 0.225$.

Mean response: standard error $\sqrt {0.6333 \times 0.225} = 0.3775$; interval $10.45 \pm 3.182 \times 0.3775 = 10.45 \pm 1.201$, that is, $(9.249,\ 11.651)$.
New observation: standard error $\sqrt {0.6333 \times 1.225} = 0.8808$; interval $10.45 \pm 3.182 \times 0.8808 = 10.45 \pm 2.803$, that is, $(7.647,\ 13.253)$.

Problem 2.13.6. For the advertising totals of Problem 2.13.2 ($n = 10$, $\hat y = 7 + x$, $S_{xx} = 60$, $S_{yy} = 110$, $\hat \sigma ^2 = 6.25$):

(a): test at the 5% level whether sales are linearly related to advertising;
(b): a manager claims that each extra $\$100$ of advertising raises mean sales by more than $500$ units. Test $H_0\colon \beta _1 = 0.5$ against $H_1\colon \beta _1 > 0.5$ at the 5% level;
(c): find a 95% confidence interval for $\sigma ^2$.

Show solution

Solution. There are $n - 2 = 8$ degrees of freedom, and $\operatorname {se}(\hat \beta _1) = \sqrt {6.25/60} = 0.3227$.

(a) $t_0 = 1/0.3227 = 3.098 > t_{0.025,8} = 2.306$: reject $H_0\colon \beta _1 = 0$. Sales are significantly related to advertising.

(b) $t_0 = (1 - 0.5)/0.3227 = 1.549$. The test is one-sided, so the critical value is $t_{0.05,8} = 1.860$. As $1.549 < 1.860$ we do not reject $H_0$: the data do not support the manager’s claim, even though the estimate $\hat \beta _1 = 1$ is above $0.5$. The estimate is too imprecise for that.

(c) $SS_{\mathrm {Res}} = 50$, $\chi ^2_{0.025,8} = 17.535$ and $\chi ^2_{0.975,8} = 2.180$, so the interval is \[ \left ( \frac {50}{17.535},\ \frac {50}{2.180} \right ) = (2.85,\ 22.94) . \] It is very wide: eight degrees of freedom tell us little about a variance.

Problem 2.13.7.

(a): Show that the confidence interval for the mean response is narrowest at $x_0 = \bar x$, and that the prediction interval at any $x_0$ is wider than the confidence interval there.
(b): As $n \to \infty $, with the $x_i$ spread so that $S_{xx} \to \infty $ and $\bar x$ stays bounded, what happens to the width of each interval?

Show solution

Solution. (a) The half-width of the confidence interval is $t_{\alpha /2,n-2} \sqrt {\hat \sigma ^2 \bigl ( \frac 1n + \frac {(x_0 - \bar x)^2}{S_{xx}} \bigr )}$. Only $(x_0 - \bar x)^2$ depends on $x_0$, and it is least, namely $0$, at $x_0 = \bar x$. The prediction interval has $1 + \frac 1n + \frac {(x_0 - \bar x)^2}{S_{xx}}$ under the square root in place of $\frac 1n + \frac {(x_0 - \bar x)^2}{S_{xx}}$, which is larger by $1$, with the same $t$ and $\hat \sigma ^2$.

(b) $\frac 1n \to 0$ and $\frac {(x_0 - \bar x)^2}{S_{xx}} \to 0$, while $\hat \sigma ^2 \to \sigma ^2$ and $t_{\alpha /2,n-2} \to z_{\alpha /2}$. So the confidence interval shrinks to the single point $E(y \mid x_0)$: with enough data the line is known exactly. The prediction interval tends to $E(y \mid x_0) \pm z_{\alpha /2}\sigma $, which does not shrink. However well the line is known, a new observation still carries its own error $\varepsilon _0$.

Problem 2.13.8. Show that $R^2$ is unchanged if every $y_i$ is replaced by $a + b y_i$ and every $x_i$ by $c + d x_i$, where $b \neq 0$ and $d \neq 0$. What happens to $\hat \beta _1$?

Show solution

Solution. Write $u_i = c + d x_i$ and $v_i = a + b y_i$. Then $\bar u = c + d\bar x$, so $u_i - \bar u = d(x_i - \bar x)$, and similarly $v_i - \bar v = b(y_i - \bar y)$. Hence \[ S_{uu} = d^2 S_{xx}, \qquad S_{vv} = b^2 S_{yy}, \qquad S_{uv} = bd\, S_{xy}, \] and by Theorem 2.12.2, \[ R^2_{\text {new}} = \frac {S_{uv}^2}{S_{uu} S_{vv}} = \frac {b^2 d^2 S_{xy}^2}{d^2 S_{xx}\, b^2 S_{yy}} = \frac {S_{xy}^2}{S_{xx} S_{yy}} = R^2 . \] The slope becomes $S_{uv}/S_{uu} = (b/d)\hat \beta _1$. Changing the units changes the slope but not how well the line fits. Measuring sales in thousands of units instead of units, for instance, divides $\hat \beta _1$ by $1000$ and leaves $R^2$ alone.

[prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.

\(x_i\)	2	4	6	8	10

\(\hat y_i\)	3.2	6.1	9.0	11.9	14.8
\(e_i\)	\(-0.2\)	0.9	\(-1.0\)	0.1	0.2


Source	SS	df	MS	\(F_0\)

Regression	84.10	1	84.10	132.79
Residual	1.90	3	0.6333

Total	86.00	4

\(x\)	2	4	6	8	10

\(y\)	3	7	8	12	15