7.4 Restricted least squares and the \(F\) test
The \(F\) statistic has a second form, which is often easier to compute and explains what the test does.
Theorem 7.4.1. The vector \(b_*\) that minimises \((Y - Xb_*)'(Y - Xb_*)\) subject to \(Rb_* = r\) is \[ b_* = b - (X'X)^{-1}R'\bigl [R(X'X)^{-1}R'\bigr ]^{-1}(Rb - r) . \] Its residuals \(e_* = Y - Xb_*\) satisfy \[ e_*'e_* - e'e = (Rb - r)'\bigl [R(X'X)^{-1}R'\bigr ]^{-1}(Rb - r) , \] so that \[ F = \frac {(\text {RSS}_R - \text {RSS}_U)/q}{\text {RSS}_U/(n - k)} , \] where \(\text {RSS}_R = e_*'e_*\) and \(\text {RSS}_U = e'e\) are the restricted and unrestricted residual sums of squares.
Proof. Minimise \(\mathcal L = (Y - Xb_*)'(Y - Xb_*) + 2\lambda '(Rb_* - r)\). The first-order condition \(-2X'(Y - Xb_*) + 2R'\lambda = 0\) gives \(b_* = b - (X'X)^{-1}R'\lambda \). Imposing \(Rb_* = r\) gives \(Rb - R(X'X)^{-1}R'\lambda = r\), so \(\lambda = [R(X'X)^{-1}R']^{-1}(Rb - r)\), which is the stated \(b_*\). The objective is convex, so this is the minimum.
For the residuals, \(e_* = Y - Xb_* = e + X(b - b_*)\). Since \(X'e = 0\), \[ e_*'e_* = e'e + (b - b_*)'X'X(b - b_*) . \] Substituting \(b - b_* = (X'X)^{-1}R'[R(X'X)^{-1}R']^{-1}(Rb - r)\), the quadratic form collapses to \((Rb - r)'[R(X'X)^{-1}R']^{-1}(Rb - r)\), because \(R(X'X)^{-1}X'X(X'X)^{-1}R' = R(X'X)^{-1}R'\). Dividing by \(q\) and by \(s^2 = \text {RSS}_U/(n - k)\) gives the \(F\) formula. □
The test therefore asks how much worse the model fits when the restrictions are imposed, measured relative to the unrestricted residual variance. If the restrictions are true, imposing them costs little; if false, the restricted residual sum of squares is much larger. In practice the restricted model is usually estimated by substituting the restrictions into it, not from the formula for \(b_*\). Under constant returns, \(\log Q = \alpha + \beta _L\log L + (1 - \beta _L)\log K + u\) rearranges to the regression of \(\log (Q/K)\) on \(\log (L/K)\).
Theorem 7.4.2. In a model with an intercept and \(k - 1\) regressors, the test of \[ H_0\colon \beta _2 = \beta _3 = \dots = \beta _k = 0 \] against the alternative that at least one slope is non-zero uses \[ F = \frac {\text {ESS}/(k - 1)}{\text {RSS}/(n - k)} = \frac {R^2/(k - 1)}{(1 - R^2)/(n - k)} \sim F_{k-1,\ n-k} \quad \text {under } H_0 . \]
Proof. Under \(H_0\) the restricted model is \(Y_i = \beta _1 + u_i\), whose least squares fit is \(\bar Y\), so \(\text {RSS}_R = \sum (Y_i - \bar Y)^2 = \text {TSS}\). Then \(\text {RSS}_R - \text {RSS}_U = \text {TSS} - \text {RSS} = \text {ESS}\), with \(q = k - 1\), and Theorem 7.4.1 gives the first form. Dividing numerator and denominator by TSS gives the second. □
This is the \(F\) statistic printed at the foot of every regression. It tests whether the regressors together explain anything, and it can reject even when no single \(t\) ratio is significant, if the regressors are highly correlated with one another. The reverse can also happen: a single significant \(t\) among many useless regressors may fail to make the overall \(F\) significant.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.