1.4 Convergence and Limit Theorems

Everything so far has been exact. Given the distribution of \(X\) we found the exact distribution of \(Y=g(X)\), of a sum, of an order statistic. That is the right ambition, and it works often enough to be worth the effort — but not always. The exact distribution of \(\bar X\) for a sample from a beta distribution has no tractable closed form, and neither does the exact distribution of most statistics anyone actually computes.

What rescues the situation is that these quantities become predictable as the sample grows. The sample mean settles onto the population mean, and the way it scatters around that mean settles onto a normal shape — whatever the population was. This section makes both statements precise. They are the bridge between the distribution theory of this chapter and the estimation theory of the next: an estimator is judged by what it does as \(n\) grows, so we need the language for that first.

1.4.1 Chebyshev’s Inequality

Everything in this section rests on one inequality, and it is worth seeing how little it assumes.

Theorem 1.4.1 (Chebyshev’s inequality). Let \(X\) be a random variable with mean \(\mu \) and finite variance \(\sigma ^2\). Then for every \(\varepsilon >0\), \[P\left (\left |X-\mu \right |\geq \varepsilon \right )\leq \frac {\sigma ^2}{\varepsilon ^2}.\] Equivalently, writing \(\varepsilon =k\sigma \), \[P\left (\left |X-\mu \right |\geq k\sigma \right )\leq \frac {1}{k^2}.\]

Proof. Take the continuous case; the discrete case replaces every integral by a sum. By definition \[\sigma ^2=\int _{-\infty }^{\infty }(x-\mu )^2f(x)\,dx \geq \int _{\left |x-\mu \right |\geq \varepsilon }(x-\mu )^2f(x)\,dx,\] because the discarded region contributes a non-negative amount. On the region that remains, \((x-\mu )^2\geq \varepsilon ^2\), so \[\sigma ^2\geq \varepsilon ^2\int _{\left |x-\mu \right |\geq \varepsilon }f(x)\,dx =\varepsilon ^2\,P\left (\left |X-\mu \right |\geq \varepsilon \right ).\] Dividing by \(\varepsilon ^2\) gives the result. □

Note 1.4.2. Notice what was not assumed: no shape, no symmetry, no named distribution — only that the variance exists. That is the inequality’s whole point, and also why it is so loose. It says at most \(\frac 14\) of any distribution lies beyond two standard deviations; for a normal distribution the true figure is about \(0.046\). A bound that holds for every distribution cannot be tight for any particular one.

Example 1.4.3. A production process has mean weight \(\mu =500\) g and standard deviation \(\sigma =8\) g, and nothing further is known about the distribution. What can be said about the proportion of items weighing between \(480\) g and \(520\) g?

Solution. The interval is \(\mu \pm 20\), that is \(\mu \pm k\sigma \) with \(k=\frac {20}{8}=2.5\). Chebyshev gives \[P\left (\left |X-500\right |\geq 20\right )\leq \frac {1}{2.5^2}=0.16,\] \[\therefore \quad P\left (480<X<520\right )\geq 1-0.16=0.84.\] At least \(84\%\) of items are within the stated range — guaranteed, whatever the shape of the distribution. If the weights were known to be normal the answer would be \(0.9876\), but that would be an extra assumption, and the whole value of the bound here is that it needs none.

1.4.2 Convergence in Probability

Definition 1.4.4. A sequence of random variables \(X_1,X_2,\ldots \) converges in probability to a constant \(c\), written \(X_n\overset {P}{\longrightarrow }c\), if for every \(\varepsilon >0\) \[\lim _{n\to \infty }P\left (\left |X_n-c\right |\geq \varepsilon \right )=0, \qquad \text {equivalently}\qquad \lim _{n\to \infty }P\left (\left |X_n-c\right |<\varepsilon \right )=1.\]

Read it as: fix any tolerance you like, however small; eventually it becomes overwhelmingly likely that \(X_n\) is within that tolerance of \(c\). It does not say \(X_n\) is ever equal to \(c\), and it does not say the sequence stops moving.

Chebyshev turns this into something checkable. If \(E(X_n)\to c\) and \(\text {Var}(X_n)\to 0\), then \[P\left (\left |X_n-c\right |\geq \varepsilon \right ) \leq \frac {E\left [(X_n-c)^2\right ]}{\varepsilon ^2}\longrightarrow 0,\] so \(X_n\overset {P}{\longrightarrow }c\). That is the standard route, and the one used throughout the next chapter: an estimator whose mean square error tends to zero is consistent, which is exactly this definition applied to \(T_n\) and \(\tau (\theta )\) — see section 2.5.

1.4.3 The Weak Law of Large Numbers

Theorem 1.4.5 (Weak law of large numbers). Let \(X_1,X_2,\ldots ,X_n\) be independent and identically distributed with mean \(\mu \) and finite variance \(\sigma ^2\). Then \[\bar X_n=\frac {1}{n}\sum _{i=1}^nX_i\ \overset {P}{\longrightarrow }\ \mu .\]

Proof. \(E\left (\bar X_n\right )=\mu \) and \(\text {Var}\left (\bar X_n\right )=\frac {\sigma ^2}{n}\). By Chebyshev, for any \(\varepsilon >0\), \[P\left (\left |\bar X_n-\mu \right |\geq \varepsilon \right ) \leq \frac {\sigma ^2}{n\varepsilon ^2}\ \longrightarrow \ 0 \quad \text {as }n\to \infty .\] □

Note 1.4.6. The law says the sample mean approaches the population mean. It does not say the sum approaches \(n\mu \) — \(\sum X_i\) has standard deviation \(\sigma \sqrt n\), which grows without bound. Dividing by \(n\) is what makes the scatter shrink, and it shrinks like \(\frac {\sigma }{\sqrt n}\): to halve the uncertainty you need four times the data. That single fact governs the cost of nearly every survey ever designed.

1.4.4 Convergence in Distribution

The law of large numbers says where \(\bar X_n\) ends up. It says nothing about the shape of its scatter on the way, and that is what the rest of this section is for. A different kind of convergence is needed, because the limit is a distribution rather than a number.

Definition 1.4.7. A sequence \(Y_1,Y_2,\ldots \) with distribution functions \(F_n\) converges in distribution to a random variable \(Y\) with distribution function \(F\), written \(Y_n\overset {d}{\longrightarrow }Y\), if \[\lim _{n\to \infty }F_n(y)=F(y)\] at every point \(y\) where \(F\) is continuous. \(F\) is then called the limiting distribution of the sequence.

Convergence in probability to a constant implies convergence in distribution to that constant; the reverse fails, since a limiting shape says nothing about where any particular \(Y_n\) has landed.

1.4.5 Finding a Limiting Distribution by Moment Generating Functions

Distribution functions are awkward to take limits of. Moment generating functions are not, and they determine the distribution, so the following is the practical tool.

Theorem 1.4.8 (Continuity theorem). Let \(Y_n\) have moment generating function \(M_n(t)\), defined for \(|t|<h\) for every \(n\). If \[\lim _{n\to \infty }M_n(t)=M(t)\qquad \text {for all }|t|<h,\] and \(M(t)\) is the moment generating function of a random variable \(Y\), then \(Y_n\overset {d}{\longrightarrow }Y\).

So a limiting distribution is found by taking the limit of a moment generating function and recognising the answer. Two limits do most of the work: \[\lim _{n\to \infty }\left (1+\frac {x}{n}\right )^n=e^x,\qquad \lim _{n\to \infty }\left (1+\frac {x}{n}+\frac {\varphi (n)}{n}\right )^n=e^x \quad \text {whenever }\varphi (n)\to 0.\] The second is the one that matters in practice: a Taylor expansion always leaves a remainder, and this says a remainder that dies faster than \(\frac 1n\) may be discarded.

Example 1.4.9. Let \(Z_n\sim \text {POI}(n)\). Find the limiting distribution of \[Y_n=\frac {Z_n-n}{\sqrt n}.\]

Solution. A Poisson variable with mean \(n\) has \(M_{Z_n}(t)=e^{n\left (e^t-1\right )}\), so \[M_{Y_n}(t)=E\left [e^{t\frac {Z_n-n}{\sqrt n}}\right ] =e^{-t\sqrt n}\,M_{Z_n}\!\left (\frac {t}{\sqrt n}\right ) =\exp \left [-t\sqrt n+n\left (e^{\frac {t}{\sqrt n}}-1\right )\right ].\] Expand the exponential inside: \[e^{\frac {t}{\sqrt n}}-1=\frac {t}{\sqrt n}+\frac {t^2}{2n} +\frac {t^3}{6n^{\frac 32}}+\cdots ,\] so \[n\left (e^{\frac {t}{\sqrt n}}-1\right )=t\sqrt n+\frac {t^2}{2} +\frac {t^3}{6\sqrt n}+\cdots \] The \(t\sqrt n\) cancels against \(-t\sqrt n\), and every remaining term after \(\frac {t^2}{2}\) carries a power of \(\frac {1}{\sqrt n}\): \[\ln M_{Y_n}(t)=\frac {t^2}{2}+O\!\left (\frac {1}{\sqrt n}\right ) \ \longrightarrow \ \frac {t^2}{2},\] \[\therefore \quad M_{Y_n}(t)\longrightarrow e^{\frac {t^2}{2}},\] which is the moment generating function of the standard normal. Hence \[\frac {Z_n-n}{\sqrt n}\ \overset {d}{\longrightarrow }\ N(0,1).\]

Note 1.4.10. The cancellation of \(t\sqrt n\) is not a lucky accident — it is what subtracting the mean is for. Dividing by \(\sqrt n\) then leaves the \(t^2\) term at exactly order one. Centre by the mean, scale by the standard deviation, and a normal limit is what falls out; that is the pattern of the next result, in general.

1.4.6 The Central Limit Theorem

Theorem 1.4.11 (Central limit theorem). Let \(X_1,X_2,\ldots ,X_n\) be independent and identically distributed with mean \(\mu \) and finite variance \(\sigma ^2>0\). Then \[Z_n=\frac {\bar X_n-\mu }{\frac {\sigma }{\sqrt n}} =\frac {\sum _{i=1}^nX_i-n\mu }{\sigma \sqrt n} \ \overset {d}{\longrightarrow }\ N(0,1).\]

Sketch. Write \(W_i=\frac {X_i-\mu }{\sigma }\), so the \(W_i\) are independent with mean \(0\) and variance \(1\), and \(Z_n=\frac {1}{\sqrt n}\sum W_i\). If \(W\) has moment generating function \(m(t)\) then \(m(0)=1\), \(m'(0)=0\) and \(m''(0)=1\), so \[m(t)=1+\frac {t^2}{2}+\varphi (t),\qquad \frac {\varphi (t)}{t^2}\to 0\ \text { as }t\to 0.\] By independence, \[M_{Z_n}(t)=\left [m\!\left (\frac {t}{\sqrt n}\right )\right ]^n =\left [1+\frac {t^2}{2n}+\varphi \!\left (\frac {t}{\sqrt n}\right )\right ]^n,\] and the remainder is of smaller order than \(\frac 1n\), so the second limit quoted above applies: \[M_{Z_n}(t)\longrightarrow e^{\frac {t^2}{2}}.\] □

Note 1.4.12. Three things the theorem does not say, each a common misreading.

It is not a statement about the observations. The \(X_i\) do not become normal, and never do; a sample from a skewed population stays skewed however large it is. It is \(\bar X\) whose distribution becomes normal.

It is not a statement about any fixed \(n\). “\(n\geq 30\)” is a rule of thumb, not part of the theorem. Thirty is generous for a symmetric population and nowhere near enough for a badly skewed one.

It needs a finite variance. Average a sample from a Cauchy distribution and \(\bar X\) has exactly the same Cauchy distribution as a single observation, no matter how large \(n\) is — averaging buys nothing at all, because the variance does not exist.

1.4.7 Normal Approximations in Practice

For a discrete variable the approximating normal density is continuous, so a probability at a single point would come out as zero. The repair is the continuity correction: give each integer the unit interval around it, replacing \(P(Y=k)\) by \(P\left (k-\frac 12<Y<k+\frac 12\right )\) and \(P(Y\leq k)\) by \(P\left (Y<k+\frac 12\right )\).

Example 1.4.13. Let \(Y\sim BIN(80,0.4)\). Use a normal approximation to find (a) \(P(Y=32)\) and (b) \(P(Y\leq 30)\).

Solution. Here \(\mu =np=32\) and \(\sigma ^2=np(1-p)=80(0.4)(0.6)=19.2\), so \(\sigma =4.3818\).

(a). With the continuity correction, \[P(Y=32)\approx P\left (31.5<Y<32.5\right ) =P\left (\frac {-0.5}{4.3818}<Z<\frac {0.5}{4.3818}\right ) =P(-0.114<Z<0.114),\] \[P(Y=32)\approx 0.5454-0.4546=0.0908,\] against an exact binomial probability of \(0.0908\).

Keep the third decimal place in \(z\) here. Rounding to \(z=\pm 0.11\) gives \(0.0876\), which is out by nearly \(4\%\) — the interval is so narrow that its width, and so the answer, is proportional to \(z\) itself.

(b). \[P(Y\leq 30)\approx P\left (Y<30.5\right ) =P\left (Z<\frac {30.5-32}{4.3818}\right )=P(Z<-0.34)=0.3669,\] against an exact value of \(0.3687\).

Note 1.4.14. Without the correction, (a) would have given \(P(Y=32)\approx 0\) and (b) \(P\left (Z<\frac {30-32}{4.3818}\right )=P(Z<-0.46)=0.3228\) — an error of nearly five percentage points on a probability of about \(0.37\). The correction costs one line and is worth it every time the variable is discrete.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.