1.5 Practice problems: distributions and transformations

Past assignment questions on moments, joint distributions and transformations. Work each one before opening the solution.

Problem 1.5.1. [Assignment 1] Prove that \[E\left (X^{(j)}\right )=(-k)^{(j)}\left (\frac {p-1}{p}\right )^j,\qquad j=1,2,3,\ldots \] if \(X\sim NB(k,p)\) with probability function \[f(x)=\binom {-k}{x}p^k(-q)^x,\qquad x=0,1,2,\ldots \] and hence find \(\text {Var}(X)\).

Show solution

Solution. Throughout, \(a^{(j)}\) is the falling factorial \(a(a-1)\cdots (a-j+1)\), so \(X^{(j)}=X(X-1)\cdots (X-j+1)\) and \(E\left (X^{(j)}\right )\) is the \(j\)th factorial moment. Note also that \(q=1-p\), so \(-q=p-1\).

Start from the negative binomial series, valid for \(|z|<1\): \[\sum _{x=0}^\infty \binom {-k}{x}z^x=(1+z)^{-k}.\] Differentiating \(j\) times term by term, \[\sum _{x=0}^\infty x^{(j)}\binom {-k}{x}z^{x-j} =\frac {d^j}{dz^j}(1+z)^{-k} =(-k)(-k-1)\cdots (-k-j+1)(1+z)^{-k-j} =(-k)^{(j)}(1+z)^{-k-j},\] and multiplying through by \(z^j\), \[\sum _{x=0}^\infty x^{(j)}\binom {-k}{x}z^x=(-k)^{(j)}z^j(1+z)^{-k-j}.\] Now put \(z=-q\) and multiply by \(p^k\), which is exactly the expectation asked for: \[E\left (X^{(j)}\right )=p^k(-k)^{(j)}(-q)^j(1-q)^{-k-j} =p^k(-k)^{(j)}(-q)^jp^{-k-j} =(-k)^{(j)}\left (\frac {-q}{p}\right )^j =(-k)^{(j)}\left (\frac {p-1}{p}\right )^j.\qquad \blacksquare \] (Taking \(j=0\) gives \(p^kp^{-k}=1\), so the probabilities do sum to one.)

For the variance, read off the first two factorial moments: \[E(X)=(-k)\left (\frac {-q}{p}\right )=\frac {kq}{p},\qquad E\left [X(X-1)\right ]=(-k)(-k-1)\left (\frac {-q}{p}\right )^2 =\frac {k(k+1)q^2}{p^2}.\] Since \(E\left (X^2\right )=E\left [X(X-1)\right ]+E(X)\), \[\text {Var}(X)=\frac {k(k+1)q^2}{p^2}+\frac {kq}{p}-\frac {k^2q^2}{p^2} =\frac {kq^2}{p^2}+\frac {kq}{p} =\frac {kq}{p^2}\left (q+p\right ) \] \[\therefore \quad \text {Var}(X)=\frac {kq}{p^2}.\]

Note 1.5.1. Factorial moments exist because \(E\left [X(X-1)\cdots \right ]\) is easier to sum than \(E\left (X^j\right )\) for the discrete distributions: the falling factorial cancels against the factorials inside the probability function. Converting back is mechanical — \(E\left (X^2\right )=E\left [X(X-1)\right ]+E(X)\), and so on.

The second half of this assignment question asks for the corresponding gamma result, \(E\left (X^k\right )=\beta ^k(k+\alpha -1)^{(k)}\). It was set again as Question 1(b) of the 2013 examination and is worked in section 4.1, so it is not repeated here.

Problem 1.5.2. [Assignment 1] Let \(X\) and \(Y\) be continuous random variables with joint p.d.f. \[f(x,y)=k(x+y),\qquad 0\leq x\leq y\leq 1.\] Find the

(a).
value of the constant \(k\);
(b).
joint c.d.f. of \(X\) and \(Y\), \(F(x,y)\);
(c).
marginal c.d.f. of \(X\) and marginal c.d.f. of \(Y\);
(d).
marginal p.d.f. of \(X\) and marginal p.d.f. of \(Y\).

Show solution

Solution. The support is the triangle \(0\leq x\leq y\leq 1\) — not the unit square. Every integral below has to respect that, and it is where most of the marks are.

(a). Integrating \(x\) from \(0\) to \(y\), then \(y\) from \(0\) to \(1\), \[1=k\int _0^1\!\!\int _0^y(x+y)\,dx\,dy =k\int _0^1\left [\frac {x^2}{2}+xy\right ]_0^y dy =k\int _0^1\frac {3y^2}{2}\,dy =\frac {k}{2},\] \[\therefore \quad k=2.\]

(b). The joint c.d.f. splits into cases according to whether the corner \((x,y)\) lies above or below the line \(u=v\).

For \(0\leq x\leq y\leq 1\) the region \(\{U\leq x,\ V\leq y\}\) meets the support in \(\{0\leq u\leq x,\ u\leq v\leq y\}\), so \[F(x,y)=\int _0^x\!\!\int _u^y2(u+v)\,dv\,du =\int _0^x\left [2uv+v^2\right ]_u^y du =\int _0^x\left (2uy+y^2-3u^2\right )du,\] \[F(x,y)=x^2y+xy^2-x^3.\]

For \(0\leq y\leq x\) with \(y\leq 1\) the constraint \(U\leq x\) is not binding, because the support already forces \(U\leq V\leq y\leq x\). So \(F(x,y)=P(V\leq y)=y^3\). Collecting the cases, \[F(x,y)=\begin {cases} 0, & x<0\ \text {or}\ y<0,\\[2pt] x^2y+xy^2-x^3, & 0\leq x\leq y\leq 1,\\[2pt] y^3, & 0\leq y\leq x,\ y\leq 1,\\[2pt] x^2+x-x^3, & 0\leq x\leq 1,\ y>1,\\[2pt] 1, & x>1,\ y>1. \end {cases}\] The two middle rows agree on \(x=y\), where both give \(y^3\), so \(F\) is continuous across the diagonal — worth checking, because a mistake in the limits almost always shows up as a jump there.

(c). Let \(y\to 1\) and \(x\to 1\) in turn: \[F_X(x)=F(x,1)=x^2+x-x^3,\quad 0\leq x\leq 1;\qquad F_Y(y)=F(1,y)=y^3,\quad 0\leq y\leq 1,\] each \(0\) below its range and \(1\) above.

(d). Differentiating the marginal c.d.f.s, \[f_X(x)=2x+1-3x^2,\quad 0\leq x\leq 1;\qquad f_Y(y)=3y^2,\quad 0\leq y\leq 1.\] The same answers come from integrating the joint density directly, \[f_X(x)=\int _x^12(x+y)\,dy=2x+1-3x^2,\qquad f_Y(y)=\int _0^y2(x+y)\,dx=3y^2,\] and both integrate to \(1\) over \([0,1]\).

Note 1.5.2. The limits are the whole question. \(f_X\) integrates \(y\) from \(x\) to \(1\), not \(0\) to \(1\), because on the support \(Y\) is never below \(X\); \(f_Y\) integrates \(x\) from \(0\) to \(y\) for the same reason. Sketch the triangle before writing anything down.

Note also that \(X\) and \(Y\) are not independent here: \(f_X(x)f_Y(y)\) is not \(2(x+y)\), and in any case a triangular support rules independence out on its own.

Problem 1.5.3. [Assignment 1] Given that \(X\) is a continuous random variable with p.d.f. \[f(x)=kx^{-(\theta +1)},\qquad x>\alpha ;\ \theta >0,\ \alpha >0,\] find the

(a).
value of the constant \(k\);
(b).
p.d.f. of \(Y=\theta \ln \left (\frac {X}{\alpha }\right )\).

Show solution

Solution. (a). Since \(\theta >0\) the integral converges at infinity: \[1=\int _\alpha ^\infty kx^{-(\theta +1)}dx =k\left [\frac {x^{-\theta }}{-\theta }\right ]_\alpha ^\infty =\frac {k\alpha ^{-\theta }}{\theta } \qquad \implies \qquad k=\theta \alpha ^\theta .\] This is the Pareto distribution with scale \(\alpha \) and shape \(\theta \).

(b). The transformation \(y=\theta \ln \left (\frac {x}{\alpha }\right )\) is strictly increasing, mapping \(x>\alpha \) onto \(y>0\), with inverse and derivative \[x=\alpha e^{\frac {y}{\theta }},\qquad \frac {dx}{dy}=\frac {\alpha }{\theta }e^{\frac {y}{\theta }}.\] Hence \[f_Y(y)=f_X\!\left (\alpha e^{\frac {y}{\theta }}\right )\left |\frac {dx}{dy}\right | =\theta \alpha ^\theta \left (\alpha e^{\frac {y}{\theta }}\right )^{-(\theta +1)} \cdot \frac {\alpha }{\theta }e^{\frac {y}{\theta }} =\alpha ^{\theta -\theta -1+1}\, e^{-\frac {(\theta +1)y}{\theta }+\frac {y}{\theta }} =e^{-y},\] \[\therefore \quad Y\sim \text {EXP}(1),\qquad y>0.\]

Note 1.5.3. Every power of \(\alpha \) and \(\theta \) cancels: the answer does not depend on either parameter. That is the point of the question. A Pareto variable is an exponential one in disguise, which is why \(\ln X\) turns Pareto data into something a \(t\)-test or a regression can handle.

Problem 1.5.4. [Assignment 1]

(a).
Let \(X\) have p.d.f. \(f(x)=\frac {x^2}{9}\), \(0<x<3\). Find the p.d.f. of \(Y=X^3\).
(b).
Let \(X\sim \text {UNIF}(0,1)\). Find the p.d.f. of (i) \(Y=e^{-X}\) and (ii) \(V=X(1-X)\).
(c).
Let \(X\) have p.d.f. \(f(x)=\frac {1}{24}x^2\), \(-2<x<4\). Find the p.d.f. of \(Y=X^2\).

Show solution

Solution. (a). \(y=x^3\) is increasing, taking \(0<x<3\) onto \(0<y<27\), with \(x=y^{\frac 13}\) and \(\frac {dx}{dy}=\frac 13y^{-\frac 23}\). So \[f_Y(y)=\frac {\left (y^{\frac 13}\right )^2}{9}\cdot \frac {1}{3}y^{-\frac 23} =\frac {y^{\frac 23}}{9}\cdot \frac {y^{-\frac 23}}{3}=\frac {1}{27}, \qquad 0<y<27,\] \[\therefore \quad Y\sim \text {UNIF}(0,27).\]

(b)(i). \(y=e^{-x}\) is decreasing, taking \(0<x<1\) onto \(e^{-1}<y<1\), with \(x=-\ln y\) and \(\left |\frac {dx}{dy}\right |=\frac {1}{y}\). Since \(f_X\equiv 1\), \[f_Y(y)=\frac {1}{y},\qquad \frac {1}{e}<y<1,\] and \(\int _{1/e}^1\frac {dy}{y}=\ln 1-\ln e^{-1}=1\), as it must.

(ii). Here \(v=x(1-x)\) is not monotone on \((0,1)\): it rises to \(\frac 14\) at \(x=\frac 12\) and falls back. The change-of-variable formula does not apply directly, so work with the distribution function.

Solving \(x^2-x+v=0\) gives the two roots \(x=\frac {1\pm \sqrt {1-4v}}{2}\), and \(X(1-X)\leq v\) holds outside them: \[F_V(v)=P\left (X\leq \frac {1-\sqrt {1-4v}}{2}\right ) +P\left (X\geq \frac {1+\sqrt {1-4v}}{2}\right ).\] As \(X\) is uniform on \((0,1)\) these probabilities are the lengths themselves: \[F_V(v)=\frac {1-\sqrt {1-4v}}{2}+1-\frac {1+\sqrt {1-4v}}{2}=1-\sqrt {1-4v}.\] Differentiating, \[f_V(v)=\frac {2}{\sqrt {1-4v}},\qquad 0<v<\frac 14,\] which integrates to \(\left [-\sqrt {1-4v}\right ]_0^{1/4}=1\).

(c). Again \(y=x^2\) is not monotone on \((-2,4)\), and this time the two branches cover different ranges. For \(0<y<4\) both \(x=\pm \sqrt y\) lie in \((-2,4)\); for \(4<y<16\) only \(x=+\sqrt y\) does, since \(-\sqrt y<-2\). With \(f_X(x)=\frac {x^2}{24}\) and \(\left |\frac {dx}{dy}\right |=\frac {1}{2\sqrt y}\), \[f_Y(y)=\begin {cases} \dfrac {2\cdot \frac {y}{24}}{2\sqrt y}=\dfrac {\sqrt y}{24}, & 0<y<4,\\[10pt] \dfrac {\frac {y}{24}}{2\sqrt y}=\dfrac {\sqrt y}{48}, & 4<y<16. \end {cases}\] As a check, \[\int _0^4\frac {\sqrt y}{24}\,dy+\int _4^{16}\frac {\sqrt y}{48}\,dy =\frac {16}{72}+\frac {112}{144}=\frac {2}{9}+\frac {7}{9}=1.\]

Note 1.5.4. Parts (b)(ii) and (c) are the whole reason this question is set. A transformation that is not one-to-one has to go through the distribution function, and the support of \(Y\) has to be worked out branch by branch — in (c) the density halves at \(y=4\), because above that point only one branch survives. Writing \(f_Y(y)=\frac {\sqrt y}{24}\) across the whole of \(0<y<16\) is the standard mistake, and it integrates to \(\frac {16}{9}\), not \(1\).

Problem 1.5.5. [Assignment 1]

(a).
Let \(X\) have the binomial p.f. \(f(x)=\binom {3}{x}\left (\frac 23\right )^x\left (\frac 13\right )^{3-x}\), \(x=0,1,2,3\). Find the p.f. of \(Y=X^2\).
(b).
Let \(X\) have p.f. \(f(x)=\frac 13\), \(x=1,2,3\). Find the p.f. of \(Y=2X+1\).
(c).
Suppose \(X\) has p.f. \(f(x)=\left (\frac 12\right )^x\), \(x=1,2,3,\ldots \) Find the p.f. of \(Y=X^3\).
(d).
If \(X\sim BIN(n,p)\), find the p.f. of \(Y=n-X\).

Show solution

Solution. For a discrete variable there is no Jacobian. When the map is one-to-one the probabilities simply move with the values: \(f_Y(y)=f_X\left (w(y)\right )\), where \(w\) is the inverse.

(a). On \(x=0,1,2,3\) the map \(y=x^2\) is one-to-one (no negative values to collide), so \(Y\) takes \(0,1,4,9\) with \(x=\sqrt y\): \[f_Y(y)=\binom {3}{\sqrt y}\left (\frac 23\right )^{\sqrt y} \left (\frac 13\right )^{3-\sqrt y},\qquad y=0,1,4,9,\] that is \[f_Y(0)=\frac {1}{27},\quad f_Y(1)=\frac {6}{27},\quad f_Y(4)=\frac {12}{27},\quad f_Y(9)=\frac {8}{27},\] which total \(\frac {27}{27}=1\).

(b). \(y=2x+1\) carries \(x=1,2,3\) to \(y=3,5,7\), and the probabilities go with them: \[f_Y(y)=\frac 13,\qquad y=3,5,7.\]

(c). \(y=x^3\) carries \(x=1,2,3,\ldots \) to the cubes \(y=1,8,27,\ldots \), with \(x=y^{\frac 13}\): \[f_Y(y)=\left (\frac 12\right )^{y^{1/3}},\qquad y=1,8,27,64,\ldots \]

(d). \(Y=n-X\) takes the values \(y=0,1,\ldots ,n\), and \[f_Y(y)=P(X=n-y)=\binom {n}{n-y}p^{n-y}q^{y}=\binom {n}{y}q^{y}p^{n-y},\] using \(\binom {n}{n-y}=\binom {n}{y}\). \[\therefore \quad Y\sim BIN(n,q)=BIN(n,1-p).\] This is obvious once stated: if \(X\) counts the successes in \(n\) trials then \(n-X\) counts the failures, and a failure has probability \(q\).

Note 1.5.5. Part (c) is printed with the support \(x=1,2,3\), which cannot be right: \(\frac 12+\frac 14+\frac 18=\frac 78\), not \(1\). The intended support is \(x=1,2,3,\ldots \), and it is taken that way above. Part (b) is genuinely finite, since \(\frac 13\) three times does sum to \(1\) — which is presumably how the misprint survived.

1.5.1 Further exercises

The remaining problems are written for these notes rather than taken from a past paper. They cover the same ground: transformations, sums and ratios, sampling distributions and order statistics.

Problem 1.5.6. [Exercise]

(a).
Let \(X\) and \(Y\) be independent \(\text {EXP}(1)\) variables. Find the p.d.f. of \(U=\frac {X}{Y}\).
(b).
Let \(X\) and \(Y\) be independent \(N\left (0,\sigma ^2\right )\) variables. Show that \(T=\frac {X}{Y}\) has the Cauchy density \(f_T(t)=\frac {1}{\pi \left (1+t^2\right )}\), whatever the value of \(\sigma \).

Show solution

Solution. (a). Put \(U=\frac {X}{Y}\) and \(V=Y\), so \(x=uv\), \(y=v\) and \[|J|=\left |\begin {matrix}v & u\\0 & 1\end {matrix}\right |=v.\] Then for \(u>0\), \(v>0\), \[f_{U,V}(u,v)=e^{-uv}e^{-v}\cdot v=v\,e^{-v(1+u)},\] \[f_U(u)=\int _0^\infty v\,e^{-v(1+u)}dv=\frac {1}{(1+u)^2},\qquad u>0,\] the integral being a gamma integral with shape \(2\) and scale \(\frac {1}{1+u}\). As a check, \(\int _0^\infty (1+u)^{-2}du=1\).

(b). By symmetry it is enough to work with the density directly. With \(T=\frac {X}{Y}\) and \(V=Y\) the same Jacobian gives \(|J|=|v|\), and \[f_{T,V}(t,v)=\frac {1}{2\pi \sigma ^2} \exp \left [-\frac {(tv)^2+v^2}{2\sigma ^2}\right ]|v|,\] so, using the symmetry in \(v\) to fold the integral onto \(v>0\), \[f_T(t)=2\int _0^\infty \frac {v}{2\pi \sigma ^2} \exp \left [-\frac {v^2\left (1+t^2\right )}{2\sigma ^2}\right ]dv.\] Substituting \(w=v^2\), \(dw=2v\,dv\), \[f_T(t)=\frac {1}{2\pi \sigma ^2}\int _0^\infty \exp \left [-\frac {w\left (1+t^2\right )}{2\sigma ^2}\right ]dw =\frac {1}{2\pi \sigma ^2}\cdot \frac {2\sigma ^2}{1+t^2} =\frac {1}{\pi \left (1+t^2\right )}.\] Every \(\sigma \) has cancelled.

Note 1.5.6. The Cauchy distribution has no mean, so it has no variance either — the integral \(\int t\,f_T(t)\,dt\) diverges at both ends. It is the standard warning that a ratio of well-behaved variables need not be well behaved: both \(X\) and \(Y\) have every moment, and their ratio has none.

Problem 1.5.7. [Exercise] Let \(X\) and \(Y\) be independent \(\text {UNIF}(0,1)\) variables. Find the p.d.f. of their product \(V=XY\).

Show solution

Solution. Put \(U=X\) and \(V=XY\), so \(x=u\), \(y=\frac {v}{u}\) and \(|J|=\frac {1}{u}\). Both original variables lie in \((0,1)\), so the image is \(0<v<u<1\), and there \[f_{U,V}(u,v)=1\cdot \frac {1}{u}=\frac {1}{u}.\] Integrating \(u\) out over \(v<u<1\), \[f_V(v)=\int _v^1\frac {du}{u}=\left [\ln u\right ]_v^1=-\ln v,\qquad 0<v<1.\] As a check, \(\int _0^1(-\ln v)\,dv=\left [v-v\ln v\right ]_0^1=1\).

Note 1.5.7. The density is unbounded as \(v\to 0^+\), yet integrates to \(1\). A density may be infinite at a point without anything going wrong — it is the area that must be finite, not the height. Note also the shape: multiplying two numbers below \(1\) usually gives something much smaller than either, which is why the mass piles up near zero.

Problem 1.5.8. [Exercise] Let \(X\sim \text {GAM}(\alpha ,1)\) and \(Y\sim \text {GAM}(\beta ,1)\) be independent. Define \[U=X+Y,\qquad V=\frac {X}{X+Y}.\] Show that \(U\sim \text {GAM}(\alpha +\beta ,1)\), that \(V\sim \text {BETA} (\alpha ,\beta )\), and that \(U\) and \(V\) are independent.

Show solution

Solution. Inverting, \(x=uv\) and \(y=u(1-v)\), with \[|J|=\left |\begin {matrix}v & u\\ 1-v & -u\end {matrix}\right | =\left |-uv-u(1-v)\right |=u.\] The image of \(\{x>0,y>0\}\) is \(\{u>0,\ 0<v<1\}\). Substituting, \[f_{U,V}(u,v)=\frac {(uv)^{\alpha -1}e^{-uv}}{\Gamma (\alpha )}\cdot \frac {\left [u(1-v)\right ]^{\beta -1}e^{-u(1-v)}}{\Gamma (\beta )}\cdot u =\frac {u^{\alpha +\beta -1}e^{-u}\,v^{\alpha -1}(1-v)^{\beta -1}} {\Gamma (\alpha )\Gamma (\beta )}.\] Multiply and divide by \(\Gamma (\alpha +\beta )\) to split it: \[f_{U,V}(u,v)= \underbrace {\frac {u^{\alpha +\beta -1}e^{-u}}{\Gamma (\alpha +\beta )}} _{\text {GAM}(\alpha +\beta ,\,1)}\cdot \underbrace {\frac {\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )} v^{\alpha -1}(1-v)^{\beta -1}}_{\text {BETA}(\alpha ,\,\beta )}.\] The joint density is a function of \(u\) alone times a function of \(v\) alone, over a rectangular region, so \(U\) and \(V\) are independent with the stated marginals. \(\blacksquare \)

Note 1.5.8. Independence here is genuinely surprising: \(V\) is built out of \(X\) and \(U\), so it is not obvious that knowing the total tells you nothing about the split. It is particular to the gamma family, and it is the reason the beta distribution turns up wherever proportions of exponential or gamma quantities are modelled.

Problem 1.5.9. [Exercise]

(a).
Let \(X\sim \text {GAM}(\alpha ,\beta )\). Show that \(Y=\frac {2X}{\beta }\sim \chi ^2(2\alpha )\).
(b).
Let \(X_1,\ldots ,X_n\) be independent \(\text {EXP}(\lambda )\) variables, each with mean \(\lambda \). Use moment generating functions to find the distribution of \(\sum _{i=1}^nX_i\), and write down its mean and variance.

Show solution

Solution. (a). The map is increasing with \(x=\frac {\beta y}{2}\) and \(\frac {dx}{dy}=\frac {\beta }{2}\), so \[f_Y(y)=\frac {1}{\Gamma (\alpha )\beta ^\alpha } \left (\frac {\beta y}{2}\right )^{\alpha -1}e^{-\frac {y}{2}}\cdot \frac {\beta }{2} =\frac {1}{2^{\alpha }\Gamma (\alpha )}y^{\alpha -1}e^{-\frac {y}{2}},\qquad y>0.\] This is the \(\chi ^2\) density with \(2\alpha \) degrees of freedom, since \(\frac {2\alpha }{2}=\alpha \). \(\blacksquare \)

(b). Each \(X_i\) has \(M(t)=(1-\lambda t)^{-1}\) for \(t<\frac 1\lambda \), so by independence \[M_{\sum X_i}(t)=\prod _{i=1}^n(1-\lambda t)^{-1}=(1-\lambda t)^{-n},\] which is the moment generating function of \(\text {GAM}(n,\lambda )\). Hence \[\sum _{i=1}^nX_i\sim \text {GAM}(n,\lambda ),\qquad E\left (\sum X_i\right )=n\lambda ,\qquad \text {Var}\left (\sum X_i\right )=n\lambda ^2.\]

Note 1.5.9. Part (a) is used constantly in chapter 3. Every exponential or gamma sum there is converted to a chi-square so that a critical value can be read from a table — \(\frac {2}{\theta _0}\sum X_i\sim \chi ^2(2n)\) is exactly (a) applied to the \(\text {GAM}(n,\theta _0)\) of (b).

Problem 1.5.10. [Exercise] Let \(Y_1\) and \(Y_2\) be independent \(N\left (\mu ,\sigma ^2\right )\) variables. Find the distribution of \[U=\frac {Y_1-3Y_2}{2}.\]

Show solution

Solution. A linear combination of independent normal variables is normal, so only the mean and variance are needed: \[E(U)=\frac {\mu -3\mu }{2}=-\mu ,\] \[\text {Var}(U)=\frac {1}{4}\left [\text {Var}(Y_1)+9\,\text {Var}(Y_2)\right ] =\frac {1}{4}\left (\sigma ^2+9\sigma ^2\right )=\frac {5\sigma ^2}{2}.\] \[\therefore \quad U\sim N\!\left (-\mu ,\ \frac {5\sigma ^2}{2}\right ).\] Formally, \(M_U(t)=M_{Y_1}\!\left (\frac {t}{2}\right ) M_{Y_2}\!\left (-\frac {3t}{2}\right )\), and multiplying the two normal moment generating functions gives \(\exp \left (-\mu t+\frac {5\sigma ^2t^2}{4}\right )\), as stated.

Note 1.5.10. The coefficients are squared in the variance but not in the mean, and the minus sign disappears there. Subtracting increases the spread just as adding does — variances of independent variables always accumulate.

Problem 1.5.11. [Exercise] Let \(X\) and \(Y\) be independent, each \(\text {EXP}(\theta )\) with density \(\frac {1}{\theta }e^{-\frac {x}{\theta }}\) on \(x>0\). Find the p.d.f. of \(U=X-Y\).

Show solution

Solution. Use the convolution formula, taking care with the support. For \(u>0\) the constraint \(y>0\) and \(x=u+y>0\) reduces to \(y>0\): \[f_U(u)=\int _0^\infty f_X(u+y)f_Y(y)\,dy =\frac {1}{\theta ^2}\int _0^\infty e^{-\frac {u+y}{\theta }}e^{-\frac {y}{\theta }}dy =\frac {e^{-\frac {u}{\theta }}}{\theta ^2}\int _0^\infty e^{-\frac {2y}{\theta }}dy =\frac {e^{-\frac {u}{\theta }}}{2\theta }.\] For \(u<0\) the roles of \(X\) and \(Y\) swap, and since they are identically distributed the answer is the mirror image. Combining, \[f_U(u)=\frac {1}{2\theta }e^{-\frac {|u|}{\theta }},\qquad -\infty <u<\infty ,\] the Laplace (double exponential) distribution. It integrates to \(2\cdot \frac {1}{2\theta }\cdot \theta =1\).

Note 1.5.11. Do the \(u>0\) case and then invoke symmetry — do not attempt both integrals. The symmetry is exact because \(X-Y\) and \(Y-X\) have the same distribution when \(X\) and \(Y\) are independent and identically distributed, and that single observation halves the work.

Problem 1.5.12. [Exercise] Let \(X\) and \(Y\) be independent \(N\left (0,\sigma ^2\right )\) variables. Find the distribution of \(W=X^2+Y^2\).

Show solution

Solution. Standardise first: \(\frac {X}{\sigma }\) and \(\frac {Y}{\sigma }\) are independent \(N(0,1)\), so \[\frac {W}{\sigma ^2}=\left (\frac {X}{\sigma }\right )^2 +\left (\frac {Y}{\sigma }\right )^2\sim \chi ^2(2),\] being a sum of two independent squared standard normals. Since \(\chi ^2(2)=\text {GAM}(1,2)\) is the exponential distribution with mean \(2\), \[W\sim \text {EXP}\left (\text {mean }2\sigma ^2\right ),\qquad f_W(w)=\frac {1}{2\sigma ^2}e^{-\frac {w}{2\sigma ^2}},\quad w>0.\]

Note 1.5.12. \(W\) is the squared distance from the origin of a point scattered symmetrically in the plane, and the result says that squared distance is exponential. It is the reason a two-dimensional Gaussian target produces exponentially distributed miss distances squared — and, read backwards, it is how normal random numbers are generated from uniform ones.

Problem 1.5.13. [Exercise] Let \(X_1,\ldots ,X_n\) be a random sample from \(N\left (\mu ,\sigma ^2\right )\).

(a).
Use moment generating functions to find the distribution of \(\bar X\), and state its mean and variance.
(b).
Show that \(\frac {\bar X-\mu }{\frac {\sigma }{\sqrt n}}\sim N(0,1)\), that \(\left (\frac {\bar X-\mu }{\frac {\sigma }{\sqrt n}}\right )^2\sim \chi ^2(1)\), and that \(\sum _{i=1}^n\left (\frac {X_i-\mu }{\sigma }\right )^2\sim \chi ^2(n)\).

Show solution

Solution. (a). Each \(X_i\) has \(M(t)=\exp \left (\mu t+\frac {\sigma ^2t^2}{2}\right )\). By independence, \[M_{\bar X}(t)=E\left [e^{\frac {t}{n}\sum X_i}\right ] =\left [M\!\left (\frac {t}{n}\right )\right ]^n =\left [\exp \left (\frac {\mu t}{n}+\frac {\sigma ^2t^2}{2n^2}\right )\right ]^n =\exp \left (\mu t+\frac {\left (\frac {\sigma ^2}{n}\right )t^2}{2}\right ),\] which is the moment generating function of a normal distribution with mean \(\mu \) and variance \(\frac {\sigma ^2}{n}\): \[\bar X\sim N\!\left (\mu ,\ \frac {\sigma ^2}{n}\right ),\qquad E\left (\bar X\right )=\mu ,\qquad \text {Var}\left (\bar X\right )=\frac {\sigma ^2}{n}.\]

(b). Standardising a normal variable gives a standard normal, and by (a) the standard deviation of \(\bar X\) is \(\frac {\sigma }{\sqrt n}\), so the first statement is immediate. The square of a standard normal is \(\chi ^2(1)\) by definition, giving the second. For the third, each \(\frac {X_i-\mu }{\sigma }\sim N(0,1)\) and they are independent, so the sum of their squares is a sum of \(n\) independent \(\chi ^2(1)\) variables, which is \(\chi ^2(n)\) because degrees of freedom add.

Note 1.5.13. Only \(\mu \) is used here, not \(\bar X\), in the third part. Replacing \(\mu \) by \(\bar X\) costs one degree of freedom and gives \(\frac {(n-1)S^2}{\sigma ^2}\sim \chi ^2(n-1)\) — the extra estimated quantity is paid for exactly once.

Problem 1.5.14. [Exercise] Let \(X_1,\ldots ,X_n\) be a random sample from \(\text {GAM}(\alpha ,\beta )\). Find the distribution of \(\bar X\), together with its mean, variance and moment generating function.

Show solution

Solution. Each \(X_i\) has \(M(t)=(1-\beta t)^{-\alpha }\), so \[M_{\bar X}(t)=\left [M\!\left (\frac {t}{n}\right )\right ]^n =\left (1-\frac {\beta t}{n}\right )^{-n\alpha },\] which is the moment generating function of a gamma distribution with shape \(n\alpha \) and scale \(\frac {\beta }{n}\): \[\bar X\sim \text {GAM}\!\left (n\alpha ,\ \frac {\beta }{n}\right ),\qquad f_{\bar X}(w)=\frac {w^{n\alpha -1}e^{-\frac {nw}{\beta }}} {\Gamma (n\alpha )\left (\frac {\beta }{n}\right )^{n\alpha }},\quad w>0.\] Its moments follow from the gamma formulas \(E=\text {shape}\times \text {scale}\) and \(\text {Var}=\text {shape}\times \text {scale}^2\): \[E\left (\bar X\right )=n\alpha \cdot \frac {\beta }{n}=\alpha \beta ,\qquad \text {Var}\left (\bar X\right )=n\alpha \left (\frac {\beta }{n}\right )^2 =\frac {\alpha \beta ^2}{n}.\]

Note 1.5.14. The mean is unchanged and the variance is divided by \(n\), exactly as for the normal case — but here the shape of \(\bar X\) changes with \(n\) as well. As \(n\) grows the shape parameter \(n\alpha \) grows, and a gamma distribution with large shape looks increasingly normal. That is the central limit theorem visible in a single formula.

Problem 1.5.15. [Exercise] Let \(X_1,\ldots ,X_6\) be a random sample from \(\text {UNIF}(0,1)\), with order statistics \(Y_1<Y_2<\cdots <Y_6\). Find the joint p.d.f. of \(Y_2\) and \(Y_5\).

Show solution

Solution. For \(i<j\) the joint density of \(Y_i\) and \(Y_j\) is \[f(y_i,y_j)=\frac {n!}{(i-1)!\,(j-i-1)!\,(n-j)!} \left [F(y_i)\right ]^{i-1} \left [F(y_j)-F(y_i)\right ]^{j-i-1} \left [1-F(y_j)\right ]^{n-j}f(y_i)f(y_j),\] the multinomial count of the ways to place \(i-1\) observations below \(y_i\), one at \(y_i\), \(j-i-1\) between, one at \(y_j\) and \(n-j\) above.

Here \(n=6\), \(i=2\), \(j=5\), and for the uniform distribution \(F(y)=y\) and \(f(y)=1\) on \((0,1)\). The constant is \[\frac {6!}{1!\,2!\,1!}=\frac {720}{2}=360,\] \[\therefore \quad f(y_2,y_5)=360\,y_2\left (y_5-y_2\right )^2\left (1-y_5\right ), \qquad 0<y_2<y_5<1.\] Integrating over the triangle confirms a total mass of \(1\).

Note 1.5.15. Each factor is doing visible work: \(y_2\) is the chance one observation falls below \(Y_2\), \((y_5-y_2)^2\) that two fall in the middle, \((1-y_5)\) that one falls above. Write the picture down first and the formula assembles itself.

Problem 1.5.16. [Exercise] Let \(X_1,\ldots ,X_n\) be a random sample from the exponential distribution with mean \(\theta \).

(a).
Show that \(Y_1=\min \left (X_1,\ldots ,X_n\right )\) is exponential with mean \(\frac {\theta }{n}\).
(b).
Find the p.d.f. of \(Y_n=\max \left (X_1,\ldots ,X_n\right )\).

Show solution

Solution. Here \(F(x)=1-e^{-\frac {x}{\theta }}\) for \(x>0\).

(a). The minimum exceeds \(y\) exactly when all \(n\) observations do: \[P\left (Y_1>y\right )=\left [1-F(y)\right ]^n =\left [e^{-\frac {y}{\theta }}\right ]^n=e^{-\frac {ny}{\theta }},\] which is the survival function of an exponential distribution with mean \(\frac {\theta }{n}\). Differentiating \(F_{Y_1}(y)=1-e^{-\frac {ny}{\theta }}\), \[f_{Y_1}(y)=\frac {n}{\theta }e^{-\frac {ny}{\theta }},\qquad y>0.\]

(b). The maximum is at most \(y\) exactly when all \(n\) are: \[F_{Y_n}(y)=\left [1-e^{-\frac {y}{\theta }}\right ]^n,\qquad f_{Y_n}(y)=\frac {n}{\theta }\left (1-e^{-\frac {y}{\theta }}\right )^{n-1} e^{-\frac {y}{\theta }},\quad y>0.\]

Note 1.5.16. The minimum stays exponential and simply speeds up by a factor of \(n\) — a consequence of the memoryless property, and the reason the first failure in a system of \(n\) components arrives \(n\) times sooner than a single one. The maximum is not exponential at all, and has no such tidy description.

Problem 1.5.17. [Exercise] Let \(X_1,\ldots ,X_n\) be a random sample from \(\text {UNIF}(0,a)\). Find the p.d.f. of the range \(R=X_{(n)}-X_{(1)}\).

Show solution

Solution. On \((0,a)\) we have \(f(x)=\frac 1a\) and \(F(x)=\frac {x}{a}\). The joint density of the minimum and maximum is \[f\left (y_1,y_n\right )=n(n-1)\left [F(y_n)-F(y_1)\right ]^{n-2}f(y_1)f(y_n) =\frac {n(n-1)\left (y_n-y_1\right )^{n-2}}{a^n},\qquad 0<y_1<y_n<a.\] Put \(R=Y_n-Y_1\) and \(S=Y_1\); then \(y_1=s\), \(y_n=s+r\), the Jacobian is \(1\), and the region becomes \(0<r<a\), \(0<s<a-r\). So \[f_R(r)=\int _0^{a-r}\frac {n(n-1)r^{n-2}}{a^n}\,ds =\frac {n(n-1)r^{n-2}(a-r)}{a^n},\qquad 0<r<a.\] It integrates to \(1\), since \[\frac {n(n-1)}{a^n}\left [\frac {a\,r^{n-1}}{n-1}-\frac {r^n}{n}\right ]_0^a =n(n-1)\left (\frac {1}{n-1}-\frac {1}{n}\right )=1.\]

Note 1.5.17. Writing \(R=\frac {r}{a}\) shows the range is \(a\) times a \(\text {BETA}(n-1,2)\) variable, whose mean is \(\frac {n-1}{n+1}\). So the sample range covers a fraction \(\frac {n-1}{n+1}\) of the interval on average — \(\frac 13\) of it for \(n=2\), but over \(90\%\) once \(n=20\). It is a usable estimator of \(a\), and the factor \(\frac {n+1}{n-1}\) is what corrects its bias.

Problem 1.5.18. [Exercise] Let \(X_1,\ldots ,X_n\) be a random sample from the \(\text {BETA}(2,3)\) distribution, with density \(f(x)=12x(1-x)^2\) on \(0<x<1\). Find the joint p.d.f. of \(Y_1\) and \(Y_n\).

Show solution

Solution. First the distribution function: \[F(x)=\int _0^x12w(1-w)^2dw =12\int _0^x\left (w-2w^2+w^3\right )dw =12\left [\frac {x^2}{2}-\frac {2x^3}{3}+\frac {x^4}{4}\right ],\] \[F(x)=6x^2-8x^3+3x^4=x^2\left (3x^2-8x+6\right ),\qquad 0<x<1,\] and \(F(1)=6-8+3=1\), as it must be.

The joint density of the extremes is the general formula with \(i=1\), \(j=n\): \[f\left (y_1,y_n\right )=n(n-1)\left [F(y_n)-F(y_1)\right ]^{n-2}f(y_1)f(y_n),\] \[\therefore \quad f\left (y_1,y_n\right )= 144\,n(n-1)\,y_1\left (1-y_1\right )^2y_n\left (1-y_n\right )^2 \left [y_n^2\left (3y_n^2-8y_n+6\right )-y_1^2\left (3y_1^2-8y_1+6\right )\right ]^{n-2}\] for \(0<y_1<y_n<1\), the \(144\) being \(12\times 12\) from the two densities.

Note 1.5.18. Nothing here depends on the distribution being beta. The formula for the extremes is the same for every continuous population; only \(F\) and \(f\) change. Getting \(F\) right is the whole job, which is why it is worth the check \(F(1)=1\).

Problem 1.5.19. [Exercise] Let \(X_1,X_2\) be a random sample of size \(2\) from the distribution with density \(f(x)=2x\), \(0<x<1\), and let \(Y_1<Y_2\) be the order statistics. Show that \[U=\frac {Y_1}{Y_2}\qquad \text {and}\qquad V=Y_2\] are independent, and identify their distributions.

Show solution

Solution. The joint density of the order statistics from a sample of size \(2\) is \(2!\,f(y_1)f(y_2)\) on \(y_1<y_2\): \[f\left (y_1,y_2\right )=2\left (2y_1\right )\left (2y_2\right )=8y_1y_2, \qquad 0<y_1<y_2<1.\] With \(u=\frac {y_1}{y_2}\) and \(v=y_2\) the inverse is \(y_1=uv\), \(y_2=v\), so \[|J|=\left |\begin {matrix}v & u\\0 & 1\end {matrix}\right |=v,\] and the region \(0<y_1<y_2<1\) becomes the rectangle \(0<u<1\), \(0<v<1\). Hence \[f_{U,V}(u,v)=8(uv)(v)\cdot v=8uv^3 =\underbrace {\left (2u\right )}_{0<u<1}\cdot \underbrace {\left (4v^3\right )}_{0<v<1}.\] The joint density factorises over a rectangle, so \(U\) and \(V\) are independent, with \[f_U(u)=2u\quad (0<u<1),\qquad f_V(v)=4v^3\quad (0<v<1),\] each integrating to \(1\).

Note 1.5.19. The rectangle matters as much as the factorisation. Had the region stayed triangular, no amount of algebra would give independence, because the range of one variable would still depend on the other. Choosing \(U\) as a ratio is what straightens the region out.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.