1.4 The conditional expectation function
A scatter plot is a sample. Behind it stands a population joint distribution of \((X, Y)\), and the object that regression analysis estimates is a feature of that distribution. It is worth being precise about which feature, because the answer explains both what regression can do without any assumptions and what it needs assumptions for.
Definition 1.4.1. Let \((X, Y)\) be random variables with \(E(Y^2) < \infty \). The conditional expectation function (CEF) of \(Y\) given \(X\) is \(m(x) = E(Y \mid X = x)\). The random variable \(m(X) = E(Y \mid X)\) is the CEF evaluated at \(X\).
The proofs below use two properties of conditional expectation, which hold whenever the expectations exist: the law of iterated expectations \(E\bigl [E(Y \mid X)\bigr ] = E(Y)\), and the fact that a function of \(X\) can be taken outside a conditional expectation given \(X\), \(E\bigl [h(X) Y \mid X\bigr ] = h(X)\, E(Y \mid X)\).
Theorem 1.4.2 (CEF decomposition). Every \(Y\) with finite variance can be written \[ Y = m(X) + \varepsilon , \qquad \varepsilon = Y - E(Y \mid X), \] where
- (i)
- \(E(\varepsilon \mid X) = 0\), and hence \(E(\varepsilon ) = 0\);
- (ii)
- \(\varepsilon \) is uncorrelated with every function \(h(X)\) for which \(E\bigl [h(X)^2\bigr ] < \infty \).
Proof. (i) \(E(\varepsilon \mid X) = E(Y \mid X) - E\bigl [m(X) \mid X\bigr ] = m(X) - m(X) = 0\). Taking expectations, \(E(\varepsilon ) = E\bigl [E(\varepsilon \mid X)\bigr ] = 0\).
(ii) By iterated expectations and (i), \[ E\bigl [h(X)\varepsilon \bigr ] = E\Bigl [E\bigl (h(X)\varepsilon \mid X\bigr )\Bigr ] = E\bigl [h(X)\, E(\varepsilon \mid X)\bigr ] = 0 . \] Since \(E(\varepsilon ) = 0\), \(\Cov \bigl (h(X), \varepsilon \bigr ) = E\bigl [h(X)\varepsilon \bigr ] = 0\). □
The theorem says something striking: the decomposition into “systematic part” and “disturbance with conditional mean zero” is not an assumption about the world. It is always available, by construction. What is an assumption is that \(m(x)\) has a particular form, say \(\alpha + \beta x\), and, for causal work, that \(m\) describes what happens when \(X\) is changed rather than merely what is observed alongside it.
Theorem 1.4.3 (The CEF is the best predictor). Among all functions \(g\) with \(E\bigl [g(X)^2\bigr ] < \infty \), the mean squared prediction error \(E\bigl [\bigl (Y - g(X)\bigr )^2\bigr ]\) is minimised by \(g = m\).
Proof. Write \(Y - g(X) = \bigl (Y - m(X)\bigr ) + \bigl (m(X) - g(X)\bigr ) = \varepsilon + h(X)\), where \(h = m - g\) is a function of \(X\). Then \begin {align*} E\bigl [\bigl (Y - g(X)\bigr )^2\bigr ] &= E(\varepsilon ^2) + 2E\bigl [h(X)\varepsilon \bigr ] + E\bigl [h(X)^2\bigr ]\\ &= E(\varepsilon ^2) + E\bigl [h(X)^2\bigr ], \end {align*}
because the cross term is zero by Theorem 1.4.2(ii). The first term does not depend on \(g\) and the second is non-negative and zero when \(g = m\). □
The CEF may be any function at all. If attention is restricted to linear predictors, the best one has an explicit formula.
Theorem 1.4.4 (Best linear predictor). Let \(\Var (X) > 0\). The values of \(a\) and \(b\) minimising \(E\bigl [(Y - a - bX)^2\bigr ]\) are \[ b^* = \frac {\Cov (X, Y)}{\Var (X)}, \qquad a^* = E(Y) - b^* E(X) . \] The error \(v = Y - a^* - b^* X\) satisfies \(E(v) = 0\) and \(\Cov (X, v) = 0\).
Proof. Let \(\mu _X = E(X)\) and \(\mu _Y = E(Y)\). The objective \(S(a, b) = E\bigl [(Y - a - bX)^2\bigr ]\) is a convex quadratic in \((a, b)\), so its minimum is where both partial derivatives vanish: \[ \frac {\partial S}{\partial a} = -2E(Y - a - bX) = 0, \qquad \frac {\partial S}{\partial b} = -2E\bigl [X(Y - a - bX)\bigr ] = 0 . \] The first gives \(a = \mu _Y - b\mu _X\), which is the statement \(E(v) = 0\). Substituting into the second, \[ E\bigl [X\bigl ((Y - \mu _Y) - b(X - \mu _X)\bigr )\bigr ] = \Cov (X, Y) - b\Var (X) = 0 , \] since \(E\bigl [X(Y - \mu _Y)\bigr ] = \Cov (X, Y)\) and \(E\bigl [X(X - \mu _X)\bigr ] = \Var (X)\). Hence \(b = b^*\), and the second equation, with \(E(v) = 0\), is \(\Cov (X, v) = 0\). Convexity follows from the Hessian \(2\begin {pmatrix} 1 & \mu _X \\ \mu _X & E(X^2)\end {pmatrix}\), whose determinant \(4\Var (X)\) is positive. □
Corollary 1.4.5. If the CEF is linear, \(m(x) = \alpha + \beta x\), then \(\alpha = a^*\) and \(\beta = b^*\): the best linear predictor and the CEF coincide.
Proof. If \(m\) is linear then \(Y = \alpha + \beta X + \varepsilon \) with \(E(\varepsilon ) = 0\) and \(\Cov (X, \varepsilon ) = 0\), by Theorem 1.4.2. These are exactly the two conditions that characterise \((a^*, b^*)\) in the proof of Theorem 1.4.4, and the solution of those conditions is unique because \(\Var (X) > 0\). □
The bivariate normal distribution is the leading case of a linear CEF: if \((X, Y)\) is bivariate normal then \(E(Y \mid X = x) = \mu _Y + \rho \frac {\sigma _Y}{\sigma _X}(x - \mu _X)\), a straight line with slope \(\rho \sigma _Y/\sigma _X = \Cov (X, Y)/\Var (X) = b^*\).
Example 1.4.6. Let \(X\) be uniform on \((0, 1)\) and \(Y = X^2 + \varepsilon \), with \(\varepsilon \) independent of \(X\), \(E(\varepsilon ) = 0\). The CEF is \(m(x) = x^2\), a curve. The moments of the uniform distribution are \(E(X^k) = \frac {1}{k+1}\), so \begin {align*} \Var (X) &= \tfrac 13 - \tfrac 14 = \tfrac {1}{12},\\ \Cov (X, Y) &= \Cov (X, X^2) = E(X^3) - E(X)E(X^2) = \tfrac 14 - \tfrac 12\cdot \tfrac 13 = \tfrac {1}{12} . \end {align*}
Hence \(b^* = 1\) and \(a^* = E(Y) - b^*E(X) = \frac 13 - \frac 12 = -\frac 16\). The best linear predictor is \(-\frac 16 + x\). It is the straight line that fits the parabola \(x^2\) best on average over \((0, 1)\). It lies above the parabola in the middle of the interval and below it at both ends, crossing it where \(x^2 = x - \frac 16\), at \(x = \frac 12 \pm \frac {1}{2\sqrt 3}\), about \(0.211\) and \(0.789\).
Least squares applied to a large sample from this population estimates \(-\frac 16 + x\). That is a correct estimate of the best linear predictor. It is a wrong estimate of the CEF, and the slope \(1\) is not “the effect of \(X\)”: the true derivative \(2x\) ranges from \(0\) to \(2\).
The lesson for the rest of the course is that least squares always estimates something well defined, the best linear predictor. Whether that something is the CEF depends on the functional form, and whether it is a causal effect depends on the assumptions about the disturbance.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.