9.1 A qualitative regressor with two levels

Suppose the response \(y\) depends on a quantitative regressor \(x\) and on which of two groups, \(A\) or \(B\), an observation comes from. Define \[ z = \begin {cases} 0 & \text {for an observation from group } A, \\ 1 & \text {for an observation from group } B. \end {cases} \] Three models are natural, and Figure 17 shows them.

(i): Coincident lines, \(y = \beta _0 + \beta _1x + \varepsilon \): the group makes no difference.
(ii): Parallel lines, \(y = \beta _0 + \beta _1x + \beta _2z + \varepsilon \). For group \(A\) (\(z = 0\)) the mean is \(\beta _0 + \beta _1x\); for group \(B\) (\(z = 1\)) it is \((\beta _0 + \beta _2) + \beta _1x\). The lines have the same slope, and \(\beta _2\) is the vertical distance between them: the difference between the groups at any fixed \(x\).
(iii): Separate lines, \(y = \beta _0 + \beta _1x + \beta _2z + \beta _3xz + \varepsilon \). For group \(A\) the mean is \(\beta _0 + \beta _1x\); for group \(B\) it is \((\beta _0 + \beta _2) + (\beta _1 + \beta _3)x\). Now \(\beta _3\) is the difference in slopes. The product \(xz\) is an interaction term: the effect of \(x\) depends on the group, and the difference between the groups depends on \(x\).

1Cβ2Pβ3Sao2a3elirpln=a=a βcβl0rji3la frd =eteeleen0t — Figure 17: Three models for one quantitative regressor \(x\) and a two-level factor. Solid: group \(A\) (\(z = 0\)). Dashed: group \(B\) (\(z = 1\)). In panel 1 the two lines are the same line.

One regressor \(z\) is enough for two groups. A second indicator, \(z' = 1 - z\) for group \(A\), would be a mistake.

Theorem 9.1.1 (The dummy-variable trap). If a factor has \(a\) levels and the model contains an intercept and an indicator for every one of the \(a\) levels, then \(\mathbf {X}\) does not have full column rank, and \(\mathbf {X}^\top \mathbf {X}\) is singular.

Proof. Every observation belongs to exactly one level, so the \(a\) indicator columns add up to the column of ones. That is a linear dependence among the columns of \(\mathbf {X}\), so \(\mathbf {Xv} = \mathbf {0}\) for the non-zero \(\mathbf {v} = (1, -1, \dots , -1, 0, \dots , 0)^\top \), and then \(\mathbf {X}^\top \mathbf {X}\mathbf {v} = \mathbf {0}\). □

So a factor with \(a\) levels is represented by \(a - 1\) indicators. The level with no indicator is the reference level, and each indicator’s coefficient is the difference between its level and the reference. Which level is the reference is a matter of convenience: the fitted values are the same whichever is chosen (Problem 9.5.1).

[next] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.