2.4 Limiting Distributions (Review)

Definition 2.4.1 (convergence in probability). A sequence of random variables \(X_1, \, \cdots \, , \, X_n\) converges in probability to a constant \(C\) if for each \(\varepsilon > 0\) \[\lim _{n\rightarrow \infty } P\left (\left |X_n - C\right | > \varepsilon \right ) = 0\] we write \(X_n \underset {P}{\longrightarrow } C\).

Theorem 2.4.2 (Chebyshev’s Inequality). Suppose \(X\) is a random variable with \(E(X) = \mu \) and \(\var (X) = \sigma ^2 < \infty \), then for any \(K > 0\) \[P\left (\left |X - \mu \right | \geq K\right ) \leq \frac {\sigma ^2}{K^2}.\]

Proof. Let \(A = \left \{\,\left |X-\mu \right | \geq K \,\right \}\) and let \(I_A\) be its indicator. On \(A\) we have \((X-\mu )^2 \geq K^2\), and off \(A\) the quantity \(K^2 I_A\) is zero while \((X-\mu )^2\) is not negative. Hence pointwise \[(X-\mu )^2 \;\geq \; K^2 I_A .\] Taking expectations preserves the inequality, so \[\sigma ^2 = E\left [(X-\mu )^2\right ] \;\geq \; K^2\, E\left [I_A\right ] = K^2\, P\left (\left |X-\mu \right | \geq K\right ),\] and dividing by \(K^2>0\) gives the result. □

Remark. The inequality cannot be strengthened to a strict inequality: for the random variable taking the values \(\mu \pm K\) each with probability \(\sigma ^2/(2K^2)\) and the value \(\mu \) otherwise, both sides are equal. Nor does it need any assumption beyond a finite variance, which is exactly why it is weak — and why the bound is useful for proving limits rather than for computing probabilities.

Theorem 2.4.3 (Weak Law of Large Numbers). Suppose \(X_1, \, \cdots \, , \, X_n\) is a random sample from a distribution with \(E(X_i) = \mu \) and \(\var (X_i) = \sigma ^2 < \infty \) then \[\overline {X_n} = \frac {1}{n}\sum ^n_{i = 1} X_i \underset {P}{\longrightarrow } \mu .\]

Proof. Because the \(X_i\) are independent with common mean \(\mu \) and common variance \(\sigma ^2\), \[E\left (\overline {X_n}\right ) = \mu ,\qquad \var \left (\overline {X_n}\right ) = \frac {1}{n^2}\sum _{i=1}^{n}\var (X_i) = \frac {\sigma ^2}{n}.\] Apply Chebyshev’s inequality to \(\overline {X_n}\): for any \(\varepsilon >0\), \[P\left (\left |\overline {X_n}-\mu \right | \geq \varepsilon \right ) \;\leq \; \frac {\var \left (\overline {X_n}\right )}{\varepsilon ^{2}} = \frac {\sigma ^{2}}{n\,\varepsilon ^{2}} \;\longrightarrow \; 0 \qquad \text {as } n\rightarrow \infty ,\] which is the definition of \(\overline {X_n}\underset {P}{\longrightarrow }\mu \). □

Note. Independence is used only to add the variances; it is not needed in full. The same argument works whenever the \(X_i\) are uncorrelated, and a version survives even when \(\var (X_i)\) is infinite, though Chebyshev’s inequality is then no longer the tool.

Definition 2.4.4 (Convergence in distribution). Let \(X_n\) be a sequence of random variables. We say \(X_n\) converges is distribution to a random variable \(X\) if \[\lim _{n\rightarrow \infty }P\left (X_n \leq x\right ) = P(X \leq x)\] for all values of \(x\) at which the right hand side \(\, F(x) = P(X\leq x)\) is continuous. We write \(X_n \underset {D}{\longrightarrow } X\).

Theorem 2.4.5 (Central Limit Theorem). Suppose \(X_1, \, \cdots \, , \, X_n\) is a random sample from a distribution with \(E(X_i) = \mu \) and \(\var (X_i) = \sigma ^2 < \infty \). Then \[Y_n = \frac {\sum ^n_{i = 1} X_i - n\mu }{\sqrt {n\, \sigma ^2}} = \frac {\sqrt {n}\left (\overline {X} - \mu \right )}{\sigma } \, \underset {D}{\longrightarrow } Z \sim N(0,1).\]

Proof. Assume the moment generating function \(m(t)=E\left (e^{tX}\right )\) exists in a neighbourhood of the origin; this is more than the theorem needs but it makes the argument elementary. Put \[W_i = \frac {X_i-\mu }{\sigma },\qquad \text {so}\qquad E(W_i)=0,\quad \var (W_i)=1,\quad Y_n = \frac {1}{\sqrt {n}}\sum _{i=1}^{n}W_i .\] Let \(m_W(t)=E\left (e^{tW}\right )\). Then \(m_W(0)=1\), \(m_W'(0)=E(W)=0\) and \(m_W''(0)=E(W^2)=1\), so the second-order Taylor expansion about the origin is \[m_W(t) = 1 + \frac {t^2}{2} + o\left (t^2\right )\qquad \text {as } t\rightarrow 0 .\] By independence the moment generating function of \(Y_n\) factorises: \[m_{Y_n}(t) = \left [m_W\!\left (\frac {t}{\sqrt {n}}\right )\right ]^{n} = \left [1 + \frac {t^{2}}{2n} + o\!\left (\frac {1}{n}\right )\right ]^{n}.\] Taking logarithms and using \(\log (1+u) = u + O(u^{2})\), \[\log m_{Y_n}(t) = n\left [\frac {t^{2}}{2n} + o\!\left (\frac {1}{n}\right )\right ] \;\longrightarrow \; \frac {t^{2}}{2},\] so \(m_{Y_n}(t) \rightarrow e^{t^{2}/2}\) for each fixed \(t\). That limit is the moment generating function of the standard normal distribution, and convergence of moment generating functions on a neighbourhood of the origin implies convergence in distribution. Hence \(Y_n \underset {D}{\longrightarrow } Z\sim N(0,1)\). □

Remark. The theorem is true without any moment generating function — finite variance alone suffices — but the general proof replaces \(m\) by the characteristic function \(\varphi (t)=E\left (e^{itX}\right )\), which always exists, and appeals to Lévy’s continuity theorem in place of the last step. The structure of the argument is identical.

Note also what is not claimed. \(Y_n\) does not converge in probability to anything: its distribution settles down, its value does not. This is the distinction the previous section drew, and the central limit theorem is the clearest place where it matters.

Theorem 2.4.6 (Limit Theorems).

1.

If \(X_n \underset {P}{\longrightarrow } a\) where \(a\) is a constant and \(g\) is a real-valued function which is continuous at \(a\), then \[g(X_n) \underset {P}{\longrightarrow } g(a).\]

2.

If \(X_n \underset {D}{\longrightarrow } X\) and \(g\) is a real-valued continuous function, then \[g(X_n) \underset {D}{\longrightarrow } g(X).\]

3.

If \(X_n \underset {P}{\longrightarrow } X\) then \(X_n \underset {D}{\longrightarrow } X\).

4.

(Slutsky) If \(X_n \underset {P}{\longrightarrow } a\) and \(Y_n \underset {D}{\longrightarrow } Y\), then

(a): \(X_n + Y_n \, \underset {D}{\longrightarrow }\, a + Y\)
(b): \(X_nY_n \, \underset {D}{\longrightarrow }\, aY\)
(c): \(\frac {Y_n}{X_n} \, \underset {D}{\longrightarrow } \, \frac {Y}{a};\hspace {0.3cm} a\neq 0\)

5.

Suppose \(a\) and \(b>0\) are constants and \[n^b\left (X_n - a\right ) \, \underset {D}{\longrightarrow }\, X.\] Let \(g\) be a real-valued function that is differentiable and whose derivative \(g'\) is continuous at \(a\), then \[n^b\, \left [g(X_n) - g(a)\right ] \, \underset {D}{\longrightarrow }\, g'(a)\, X.\]

Proof.

(1)

(3)

Let \(x\) be a continuity point of \(F\), the distribution function of \(X\), and let \(\varepsilon >0\). Splitting on whether \(\left |X_n-X\right |\) is small, \[P(X_n\leq x) \leq P(X\leq x+\varepsilon ) + P\left (\left |X_n-X\right |>\varepsilon \right ),\] \[P(X\leq x-\varepsilon ) \leq P(X_n\leq x) + P\left (\left |X_n-X\right |>\varepsilon \right ).\] Letting \(n\rightarrow \infty \) kills the last term in each line, so \[F(x-\varepsilon ) \;\leq \; \liminf _n P(X_n\leq x) \;\leq \; \limsup _n P(X_n\leq x) \;\leq \; F(x+\varepsilon ).\] Now let \(\varepsilon \downarrow 0\); continuity of \(F\) at \(x\) closes the two outer bounds onto \(F(x)\), so \(P(X_n\leq x)\rightarrow F(x)\).

(5), the delta method

Write \(Z_n = n^{b}\left (X_n-a\right )\), so \(Z_n \underset {D}{\longrightarrow } X\) by hypothesis. Since \(b>0\), \(X_n - a = n^{-b}Z_n \underset {P}{\longrightarrow } 0\). By the mean value theorem there is a point \(\xi _n\) between \(a\) and \(X_n\) with \[n^{b}\left [g(X_n)-g(a)\right ] = g'(\xi _n)\, n^{b}\left (X_n-a\right ) = g'(\xi _n)\, Z_n .\] Because \(\xi _n\) lies between \(a\) and \(X_n\) we have \(\xi _n \underset {P}{\longrightarrow } a\), and \(g'\) is continuous at \(a\), so part (1) gives \(g'(\xi _n)\underset {P}{\longrightarrow } g'(a)\). Slutsky’s theorem, part 4(b), applied to the product then yields \[n^{b}\left [g(X_n)-g(a)\right ] \underset {D}{\longrightarrow } g'(a)\,X .\]

(2) and (4)

Both are instances of the continuous mapping theorem, which rests on the portmanteau characterisation — that \(X_n\underset {D}{\longrightarrow }X\) if and only if \(E\left [h(X_n)\right ]\rightarrow E\left [h(X)\right ]\) for every bounded continuous \(h\). Given that, (2) is immediate: if \(h\) is bounded and continuous then so is \(h\circ g\). Part (4) follows by applying the same statement to the pair \((X_n,Y_n)\), which converges jointly in distribution to \((a,Y)\) precisely because the first coordinate has a constant limit; the maps \((x,y)\mapsto x+y\), \(xy\) and \(y/x\) are continuous at every point \((a,y)\), the last provided \(a\neq 0\). The portmanteau theorem itself is a result in measure theory and is taken as known here. □

Note. Part (4) is the workhorse. Almost every asymptotic argument later in this course has the shape “something converging in distribution, multiplied or divided by something converging in probability to a constant”, and Slutsky’s theorem is what licenses treating the second as though it already were that constant. The requirement that the limit be a constant is not decorative: for two sequences each converging in distribution the conclusion is false in general, because their joint behaviour is not determined by their marginals.

[next] [prev] [prev-tail] [front] [up]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.