4.3 Convergence in distribution
Consistency says where an estimator goes; to make inferences we also need to know how it is distributed around its limit in large samples.
Definition 4.3.1. \(X_n\) converges in distribution to a random variable \(X\), written \(X_n \xrightarrow {d} X\), if \(P(X_n \leq x) \to P(X \leq x)\) at every \(x\) where the distribution function of \(X\) is continuous. The distribution of \(X\) is the limiting distribution of \(X_n\).
Theorem 4.3.2 (Central limit theorem, Lindeberg–Lévy). Let \(X_1, X_2, \dots \) be independent and identically distributed with mean \(\mu \) and finite variance \(\sigma ^2 > 0\). Then \[ \sqrt n\,(\bar X_n - \mu ) \xrightarrow {d} N(0, \sigma ^2) . \]
Proof. We prove it under the extra assumption that the moment generating function of \(X_i\) exists near zero; the general proof is the same with the characteristic function in its place. Let \(W_i = (X_i - \mu )/\sigma \), with moment generating function \(M(s)\), and \(S_n = \sqrt n(\bar X_n - \mu )/\sigma = n^{-1/2}\sum W_i\). By independence, \[ M_{S_n}(t) = \bigl [M(t/\sqrt n)\bigr ]^n . \] Since \(E(W) = 0\) and \(E(W^2) = 1\), Taylor’s theorem gives \(M(s) = 1 + \frac 12s^2 + r(s)\) with \(r(s)/s^2 \to 0\) as \(s \to 0\). Hence \[ \log M_{S_n}(t) = n\log \Bigl (1 + \frac {t^2}{2n} + r(t/\sqrt n)\Bigr ) \longrightarrow \frac {t^2}{2}, \] because \(n\,r(t/\sqrt n) = t^2\,r(t/\sqrt n)/(t/\sqrt n)^2 \to 0\) and \(n\log (1 + c_n/n) \to c\) whenever \(c_n \to c\). The limit \(e^{t^2/2}\) is the moment generating function of \(N(0, 1)\), and convergence of moment generating functions on a neighbourhood of zero implies convergence in distribution (Curtiss’s continuity theorem). Multiplying by \(\sigma \) gives the result. □
The CLT is the reason the normal distribution is everywhere in econometrics. Estimators are, to a first approximation, averages, and averages are approximately normal whatever the distribution of what is averaged. Two tools transfer that normality from averages to the estimators built from them.
Theorem 4.3.3 (Slutsky). If \(X_n \xrightarrow {d} X\) and \(Y_n \xrightarrow {p} c\), a constant, then \[ X_n + Y_n \xrightarrow {d} X + c, \qquad Y_nX_n \xrightarrow {d} cX, \qquad X_n/Y_n \xrightarrow {d} X/c \ \ (c \neq 0) . \]
Proof. We prove the first statement; the others follow by similar arguments. Let \(F\) be the distribution function of \(X\), and let \(x\) be such that \(F\) is continuous at \(x - c\). For \(\varepsilon > 0\) such that \(F\) is continuous at \(x - c \pm \varepsilon \), \begin {align*} P(X_n + Y_n \leq x) &\leq P(X_n \leq x - c + \varepsilon ) + P(|Y_n - c| > \varepsilon ),\\ P(X_n + Y_n \leq x) &\geq P(X_n \leq x - c - \varepsilon ) - P(|Y_n - c| > \varepsilon ) . \end {align*}
Letting \(n \to \infty \), the limit of \(P(X_n + Y_n \leq x)\) lies between \(F(x - c - \varepsilon )\) and \(F(x - c + \varepsilon )\). Letting \(\varepsilon \to 0\) through continuity points, both bounds tend to \(F(x - c) = P(X + c \leq x)\). □
Theorem 4.3.4 (The delta method). If \(\sqrt n(T_n - \theta ) \xrightarrow {d} N(0, v)\) and \(g\) is differentiable at \(\theta \) with \(g'(\theta ) \neq 0\), then \[ \sqrt n\bigl (g(T_n) - g(\theta )\bigr ) \xrightarrow {d} N\bigl (0, g'(\theta )^2v\bigr ) . \]
Proof. First, \(T_n \xrightarrow {p} \theta \): by Slutsky, \(T_n - \theta = n^{-1/2}\cdot \sqrt n(T_n - \theta ) \xrightarrow {d} 0 \cdot N(0, v) = 0\), and convergence in distribution to a constant is convergence in probability to it. Define \(h(t) = \bigl (g(t) - g(\theta )\bigr )/(t - \theta )\) for \(t \neq \theta \) and \(h(\theta ) = g'(\theta )\). Differentiability makes \(h\) continuous at \(\theta \), so \(h(T_n) \xrightarrow {p} g'(\theta )\) by Theorem 4.2.5. Then \[ \sqrt n\bigl (g(T_n) - g(\theta )\bigr ) = h(T_n)\cdot \sqrt n(T_n - \theta ) \xrightarrow {d} g'(\theta )\,N(0, v) \] by Slutsky’s theorem. □
In practice the theorem is used as an approximation: \(\operatorname {se}\bigl (g(T_n)\bigr ) \approx |g'(T_n)|\operatorname {se}(T_n)\). This is the formula used for the growth rate \(\hat g = e^b - 1\) in Section 3.2, with \(g'(b) = e^b\).
Example 4.3.5. For the Bernoulli odds of Example 4.2.6, \(\sqrt n(\bar Y - \theta ) \xrightarrow {d} N\bigl (0, \theta (1 - \theta )\bigr )\) by the CLT. With \(g(\theta ) = \theta /(1 - \theta )\), \(g'(\theta ) = 1/(1 - \theta )^2\), so \[ \sqrt n(\hat \Upsilon - \Upsilon ) \xrightarrow {d} N\left (0,\ \frac {\theta }{(1 - \theta )^3}\right ) . \] With \(n = 400\) and \(\bar Y = 0.2\), \(\hat \Upsilon = 0.25\) and \(\operatorname {se}(\hat \Upsilon ) \approx \sqrt {0.2/(0.8^3 \times 400)} = 0.031\).
The delta method also explains the Fisher transformation used in Software 1.5.2. For a bivariate normal sample the correlation satisfies \(\sqrt n(r - \rho ) \xrightarrow {d} N\bigl (0, (1 - \rho ^2)^2\bigr )\). The variance depends on \(\rho \), which is unknown. With \(g(r) = \tanh ^{-1}r\), \(g'(\rho ) = 1/(1 - \rho ^2)\) and the limiting variance becomes \((1 - \rho ^2)^2/(1 - \rho ^2)^2 = 1\), free of \(\rho \). That is why \(\tanh ^{-1}r\) has variance close to \(1/n\), refined in practice to \(1/(n - 3)\). A function chosen to make the variance constant in this way is a variance-stabilising transformation.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.