4.2 Convergence in probability
Definition 4.2.1. A sequence of random variables \(X_n\) converges in probability to a constant \(c\), written \(X_n \xrightarrow {p} c\) or \(\plim X_n = c\), if for every \(\varepsilon > 0\) \[ \lim _{n\to \infty } P\bigl (|X_n - c| > \varepsilon \bigr ) = 0 . \] An estimator \(\hat \theta _n\) of \(\theta \) is consistent if \(\hat \theta _n \xrightarrow {p} \theta \).
Consistency says that, with a large enough sample, the estimator is close to the truth with probability as high as we like. It is the minimal requirement for an estimator. An inconsistent estimator remains wrong, by an amount that does not shrink, however much data is collected.
Lemma 4.2.2 (Markov and Chebyshev inequalities). If \(W \geq 0\) and \(a > 0\), then \(P(W \geq a) \leq E(W)/a\). Consequently, for any random variable \(X\) with finite variance and any \(\varepsilon > 0\), \[ P\bigl (|X - E(X)| \geq \varepsilon \bigr ) \leq \frac {\Var (X)}{\varepsilon ^2}, \qquad P\bigl (|X - c| \geq \varepsilon \bigr ) \leq \frac {E\bigl [(X - c)^2\bigr ]}{\varepsilon ^2} . \]
Proof. Since \(W \geq a\,\mathbf 1\{W \geq a\}\) pointwise, \(E(W) \geq aP(W \geq a)\). Apply this to \(W = (X - c)^2\) and \(a = \varepsilon ^2\), noting that \(|X - c| \geq \varepsilon \) exactly when \((X - c)^2 \geq \varepsilon ^2\); the case \(c = E(X)\) is the first inequality. □
Theorem 4.2.3 (Mean square convergence implies consistency). If \(E(\hat \theta _n) \to \theta \) and \(\Var (\hat \theta _n) \to 0\), then \(\hat \theta _n \xrightarrow {p} \theta \).
Proof. \(E\bigl [(\hat \theta _n - \theta )^2\bigr ] = \Var (\hat \theta _n) + \bigl (E(\hat \theta _n) - \theta \bigr )^2 \to 0\), and by Lemma 4.2.2, \(P\bigl (|\hat \theta _n - \theta | \geq \varepsilon \bigr ) \leq E\bigl [(\hat \theta _n - \theta )^2\bigr ]/\varepsilon ^2 \to 0\). □
Theorem 4.2.4 (Weak law of large numbers). Let \(X_1, X_2, \dots \) be uncorrelated with common mean \(\mu \) and variances bounded by a constant \(K\). Then \(\bar X_n \xrightarrow {p} \mu \).
Proof. \(E(\bar X_n) = \mu \) and \(\Var (\bar X_n) = n^{-2}\sum \Var (X_i) \leq K/n \to 0\). Apply Theorem 4.2.3. □
For independent and identically distributed variables the conclusion holds under the weaker condition that \(E|X_i| < \infty \), with no variance required (Khinchine’s theorem). Versions for dependent but stationary sequences, needed for time series, are stated in Section 4.5.
Theorem 4.2.5 (Continuous mapping). If \(X_n \xrightarrow {p} c\) and \(g\) is continuous at \(c\), then \(g(X_n) \xrightarrow {p} g(c)\). The same holds for vectors: if \(X_n \xrightarrow {p} c\) and \(Y_n \xrightarrow {p} d\) then \(X_n + Y_n \xrightarrow {p} c + d\), \(X_nY_n \xrightarrow {p} cd\), and \(X_n/Y_n \xrightarrow {p} c/d\) if \(d \neq 0\).
Proof. Given \(\varepsilon > 0\), continuity gives a \(\delta > 0\) with \(|g(x) - g(c)| < \varepsilon \) whenever \(|x - c| < \delta \). So the event \(|g(X_n) - g(c)| \geq \varepsilon \) is contained in the event \(|X_n - c| \geq \delta \), whose probability tends to zero. For two sequences, apply the same argument to \(g(x, y)\) continuous at \((c, d)\), using \(P\bigl (\|(X_n, Y_n) - (c, d)\| \geq \delta \bigr ) \leq P(|X_n - c| \geq \delta /2) + P(|Y_n - d| \geq \delta /2)\). □
This is what makes \(\plim \) so much easier to work with than expectation. \(E(1/X) \neq 1/E(X)\) in general, but \(\plim (1/X_n) = 1/\plim X_n\); and \(E(X_n/Y_n)\) is usually intractable, while \(\plim (X_n/Y_n)\) is simply the ratio of the two limits.
Example 4.2.6. Let \(Y_1, \dots , Y_n\) be a random sample from the Bernoulli distribution with \(P(Y = 1) = \theta \), \(0 < \theta < 1\).
- (i)
- \(E(\bar Y) = \theta \) and \(\Var (\bar Y) = \theta (1 - \theta )/n \to 0\), so \(\bar Y\) is consistent for \(\theta \) by Theorem 4.2.3.
- (ii)
- The odds \(\Upsilon = \theta /(1 - \theta )\) is a continuous function of \(\theta \) on \((0, 1)\), so by Theorem 4.2.5 \(\hat \Upsilon = \bar Y/(1 - \bar Y)\) is consistent for \(\Upsilon \). It is not unbiased: in a finite sample \(\bar Y = 1\) has positive probability, and then \(\hat \Upsilon \) is not even defined. Consistency survives such defects because their probability vanishes.
- (iii)
- Let \(\bar Y\) be the mean annual earnings of a sample of \(n\) workers with secondary education and \(\bar Z\) that of an independent sample of \(n\) graduates, with population means \(\mu _Y > 0\) and \(\mu _Z\). The percentage by which graduates’ mean earnings exceed secondary-educated workers’ is \(100(\mu _Z/\mu _Y - 1)\). Since \(\bar Y \xrightarrow {p} \mu _Y\) and \(\bar Z \xrightarrow {p} \mu _Z\), Theorem 4.2.5 gives \[ 100\left (\frac {\bar Z}{\bar Y} - 1\right ) \xrightarrow {p} 100\left (\frac {\mu _Z}{\mu _Y} - 1\right ), \] a consistent estimator, although \(E(\bar Z/\bar Y) \neq \mu _Z/\mu _Y\) in general.
Example 4.2.7. A random sample \(X_1, \dots , X_n\) is drawn from a population with mean \(\mu \) and variance \(\sigma ^2 > 0\). Consider \[ Z_n = \frac 12X_1 + \frac 14X_2 + \frac 18X_3 + \dots + \frac {1}{2^n}X_n . \] (i) \(E(Z_n) = \mu \sum _{i=1}^n 2^{-i} = \mu \,\dfrac {\frac 12(1 - 2^{-n})}{1 - \frac 12} = \mu (1 - 2^{-n})\), which tends to \(\mu \): \(Z_n\) is biased but asymptotically unbiased.
(ii) Is \(Z_n\) consistent? Its variance is \[ \Var (Z_n) = \sigma ^2\sum _{i=1}^n 4^{-i} = \sigma ^2\,\frac {\frac 14(1 - 4^{-n})}{1 - \frac 14} = \frac {\sigma ^2}{3}(1 - 4^{-n}) \to \frac {\sigma ^2}{3} , \] which does not tend to zero, so Theorem 4.2.3 cannot be used. That alone does not prove inconsistency: a variance that fails to vanish is compatible with convergence in probability in pathological cases. The proof uses the structure of \(Z_n\). Let \(Z_\infty = \sum _{i=1}^\infty 2^{-i}X_i\). Then \[ E\bigl [(Z_\infty - Z_n)^2\bigr ] = \Var \Bigl (\sum _{i>n}2^{-i}X_i\Bigr ) + \Bigl (\mu \sum _{i>n}2^{-i}\Bigr )^2 = \frac {\sigma ^2}{3}4^{-n} + \mu ^2 4^{-n} \to 0 , \] so \(Z_n - Z_\infty \xrightarrow {p} 0\). If also \(Z_n \xrightarrow {p} \mu \), then \(Z_\infty = Z_n + (Z_\infty - Z_n) \xrightarrow {p} \mu \); but \(Z_\infty \) does not depend on \(n\), so it would equal \(\mu \) with probability one. That is impossible, since \(\Var (Z_\infty ) = \sigma ^2/3 > 0\). Hence \(Z_n\) is not consistent.
The reason is plain from the weights. Half of the weight is on \(X_1\) alone, whatever \(n\) is, so the estimator never averages out the noise in the first observation. Consistency needs every observation’s weight to shrink to zero, as the weights \(1/n\) of \(\bar X_n\) do.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.