2.6 Some Important Continuous Probability Density Functions

In this section, we describe models frequently used in continuous random phenomena. These distributions are the building blocks of statistical inference and engineering applications.

2.6.1 The Uniform Distribution

The uniform distribution provides a probability model for selecting points at random from an \([\alpha , \beta ]\). An important application of the uniform distribution lies in random number generation.

Diagrams of the Uniform Distribution.

The mean of \(X\) is \begin {align*} E(X) &=\int ^{\beta }_{\alpha }x\frac {1}{\beta -\alpha }\, dx\\ &=\frac {1}{\beta -\alpha }\int ^{\beta }_{\alpha }x\, dx\\ &=\frac {\alpha ^2}{2(\beta -\alpha )}\Bigg |_{\alpha }^{\beta }\\ &=\frac {\beta ^2-\alpha ^2}{2(\beta -\alpha )}\\ &=\frac {(\beta +\alpha )(\beta -\alpha )}{2(\beta -\alpha )}\\ & =\frac {\beta +\alpha }{2}. \end {align*}

Also \begin {align*} E(X^2) & = \frac {1}{\beta - \alpha }\int ^{\beta }_{\alpha } x^2\,dx\\ & = \frac {1}{\beta - \alpha }\left [\frac {x^3}{3}\right ]^{\beta }_{\alpha }\\ & = \frac {1}{\beta -\alpha }\, \frac {\beta ^3-\alpha ^3}{3}\\ & = \frac {(\beta - \alpha )(\beta ^2 + \beta \alpha + \alpha ^2)}{3(\beta - \alpha )}\\ & = \frac {\beta ^2 + \beta \alpha + \alpha ^2}{3}. \end {align*}

Hence, variance is \begin {align*} \operatorname {Var}(X) & = \frac {\beta ^2 + \beta \alpha + \alpha ^2}{3}- \frac {(\beta + \alpha )^2}{4}\\ & = \frac {\beta ^2 - 2\beta \alpha + \alpha ^2}{12}\\ & = \frac {(\beta - \alpha )^2}{12}. \end {align*}

The m.g.f of \(X\) is \begin {align*} M_X(t) &=E\left (e^{tx}\right )\\ &=\int ^{\beta }_{\alpha }e^{tx}\, dx\\ &=\frac {1}{\beta -\alpha }\int ^{\beta }_{\alpha }e^{tx}\, dx\\ & =\frac {1}{\beta -\alpha }\,\frac {e^{tx}}{t}\Bigg |_{\alpha }^{\beta }\\ &=\frac {e^{\beta t}-e^{\alpha t}}{t(\beta -\alpha )}. \end {align*}

Thus \[M_X(t)= \begin {cases} \frac {e^{\beta t}-e^{\alpha t}}{t(\beta -\alpha )}\, , & \text {if}\,\, t \neq 0\\ 1, & \text {if}\,\, t = 0.\\ \end {cases}.\]

2.6.2 Exponential Distribution

The exponential is the distribution of the amount of time until some specific event occurs (i.e. waiting time). For instance, how long one waits before a new component “dies”?

Graphs of an Exponential

The cumulative distribution function is \begin {align*} F_X(x) & = P[X \leq x]\\ & = \int ^x_0 \lambda \, e^{-\lambda x}\, dx\\ & = -e^{-\lambda x}\Big |^x_0\\ & = 1 - e^{-\lambda x}\, , \quad x \geq 0. \end {align*}

The moment generating function is \begin {align*} M_X(t) & = E\left [e^{tX}\right ]\\ & = \int ^{\infty }_0 e^{tx}\, \lambda e^{-\lambda x}\, dx\\ & = \lambda \int ^{\infty }_0 e^{-(\lambda - t)x}\, dx\\ & = \frac {\lambda }{\lambda - t}\,, \quad t < \lambda . \end {align*}

Then differentiating \(M'_X(t)\) we get \[M'_X(t) = \frac {\lambda (-1) (-1)}{(\lambda - t)^2} = \frac {\lambda }{(\lambda - t})^2\] \[M''_X(t) = \frac {\lambda (-1)(-2)}{(\lambda - t)^3} = \frac {2\lambda }{(\lambda - t)^3}\] Then the mean is \[E(X) = M'_X(0) = \frac {\lambda }{\lambda ^2} = \frac {1}{\lambda }\] and \[E(X^2) = M''_X(0) = \frac {2\lambda }{\lambda ^3} = \frac {2}{\lambda ^2}\] then variance \[\operatorname {Var}(X) = \frac {2}{\lambda ^2}-\left (\frac {1}{\lambda ^2}\right ) = \frac {1}{\lambda ^2}.\]

2.6.3 Gamma Distribution

Properties of \(\Gamma \)

(i).

\(\Gamma (\alpha + 1) = \alpha \,\Gamma (\alpha )\)

(ii).

\(\Gamma (n) = (n-1)!, \, n = 1, 2, \cdots \)

(iii).

\(\Gamma \left (\frac {1}{2}\right ) = \sqrt {\pi }\)

The moment generating function of \(X\) is \begin {align*} M_X(t) &=E\left [e^{tx}\right ] =\int ^{\infty }_{0} e^{tx}\,\frac {\lambda ^{\alpha }\, x^{\alpha - 1}\, e^{-\alpha \lambda }}{\Gamma (\alpha )}\, dx\\ & = \frac {\alpha ^{\lambda }}{\Gamma (\alpha )}\int ^{\infty }_0x^{\alpha - 1}\, e^{-(\lambda - t)x}\, dx\\ & =\frac {\lambda ^{\alpha }}{\Gamma (\alpha )}\int ^{\infty }_0\left (\frac {y}{\lambda -t}\right )^{\alpha -1}e^{-y}\, \frac {dy}{\lambda -t},\quad \quad \quad \text {letting}\quad y = (\lambda - t)x\\ &=\frac {\lambda ^{\alpha }}{\Gamma (\alpha )}\frac {1}{(\lambda -t)^{\alpha -1}}\,\frac {1}{(\lambda - t)}\int ^{\infty }_{0}y^{\alpha -1}\, e^{-y}\, dy\\ &=\frac {\lambda ^{\alpha }}{\Gamma (\alpha )}\quad \left (\frac {1}{\lambda -t}\right )^{\alpha }\quad \Gamma (\alpha )\\ &=\left (\frac {\lambda }{\lambda -t}\right )^{\alpha }. \end {align*}

Differentiating \(M_X(t)\) \begin {align*} M'_X(t) & = \lambda ^{\alpha }(-\alpha )(\lambda -t)^{-\alpha -1}(-1) = \alpha \lambda ^{\alpha }(\lambda -t)^{-\alpha -1} = \frac {\alpha \, \lambda ^{\alpha }}{(\lambda - t)^{\alpha + 1}}\\ M''_X(t) & = \frac {\alpha \, [-(\alpha + 1)](-1)\,\lambda ^{\alpha }}{(\lambda - t)^{\alpha + 2}}=\frac {\alpha (\alpha + 1) \lambda ^{\alpha }}{(\lambda - t)^{\alpha + 2}} \end {align*}

Then the mean is \[E(X) = M'_X(0) = \frac {\alpha \, \lambda ^{\alpha }}{\lambda ^{\alpha + 1}}= \frac {\alpha }{\lambda }\] and for the variance \[E(X^2) = M''_X(0) = \frac {\alpha (\alpha + 1) \, \lambda ^{\alpha }}{\lambda ^{\alpha + 2}} = \frac {\alpha (\alpha + 1)}{\lambda ^{2}}\] therefore \[\operatorname {Var}(X) = \frac {\alpha ^2 + \alpha }{\lambda ^2} - \left (\frac {\alpha }{\lambda }\right )= \frac {\alpha }{\lambda ^2}.\]

Graphs of the Gamma

2.6.4 Beta Distribution

Properties of \(B\):

1.

\(B(\alpha , \beta ) = \frac {\Gamma (\alpha )\, \Gamma (\beta )}{\Gamma (\alpha + \beta )}\)

2.

\(B(\alpha , \beta ) = B(\beta , \alpha )\)

3.

\(B(\alpha + 1, \beta ) = \frac {\alpha }{\alpha + \beta }\, B(\alpha , \beta )\)

4.

\(B(\alpha , \beta + 1) = \frac {\beta }{\alpha + \beta }\, B(\alpha ,\beta )\)

The \(k^{\text {th}}\) expectation of \(X\) is \begin {align*} E(x^k) & = \int \limits _x x^kf_X(x)\,dx\\ & =\int ^1_0\frac {x^k\, x^{\alpha -1}\, (1-x)^{\beta -1}}{B(\alpha ,\beta )}\,dx\\ & = \frac {1}{B(\alpha ,\beta )}\int ^1_0x^{k+\alpha -1}(1-x)^{\beta -1}\,dx\\ & = \frac {1}{B(\alpha ,\beta )}B(k+\alpha ,\beta )\\ & = \frac {\frac {\Gamma (k+\alpha )\Gamma (\beta )}{\Gamma (k+\alpha +\beta )}}{\frac {\Gamma (\alpha )\Gamma (\beta )}{\Gamma (\alpha +\beta )}} =\frac {\Gamma (k+\alpha )\Gamma (\beta )}{\Gamma (k+\alpha +\beta )}\times \frac {\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}\\ & = \frac {\Gamma (k+\alpha )\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (k+\alpha +\beta )}. \end {align*}

When \(k = 1\), we get the mean of \(X\) \[E(x)=\frac {\Gamma (\alpha +1)\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\alpha +\beta +1)}=\frac {\alpha \Gamma (\alpha )\Gamma (\alpha +\beta )}{\Gamma (\alpha )(\alpha +\beta )\Gamma (\alpha +\beta )}=\frac {\alpha }{\alpha +\beta }.\]

When \(k = 2\), we get \begin {align*} E(x^2) & = \frac {\Gamma (\alpha +2)\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\alpha +\beta +2)}\\ & = \frac {(\alpha +1)\alpha \Gamma (\alpha )\Gamma (\alpha +\beta )}{\Gamma (\alpha )(\alpha +\beta +1)(\alpha +\beta )\Gamma (\alpha +\beta )}\\ & = \frac {\alpha (\alpha +1)}{(\alpha +\beta +1)(\alpha +\beta )}. \end {align*}

Therefore, the variance of \(X\) is \begin {align*} \operatorname {Var}(x) & = E(x^2)-(E(x))^2\\ &=\frac {\alpha (\alpha +1)}{(\alpha +\beta )(\alpha +\beta +1)}-\left (\frac {\alpha }{\alpha +\beta }\right )^2\\ & = \frac {\alpha }{\alpha +\beta }\left [\frac {\alpha +1}{\alpha +\beta +1}-\frac {\alpha }{\alpha +\beta }\right ]\\ & = \frac {\alpha }{\alpha +\beta }\left [\frac {(\alpha +1)(\alpha +\beta )-\alpha (\alpha +\beta +1)}{(\alpha +\beta )(\alpha +\beta +1)}\right ]\\ & = \frac {\alpha }{(\alpha +\beta )^2}\left [\frac {\alpha ^2+\alpha \beta +\alpha +\beta -\alpha ^2-\alpha \beta -\alpha }{\alpha +\beta +1}\right ]\\ & = \frac {\alpha \beta }{(\alpha +\beta )^2(\alpha +\beta +1)}. \end {align*}

Graph of the Beta Distribution.

2.6.5 Normal Distribution

The main importance of the normal distribution lies in the central limit theorem, which is what makes it turn up in problems that have nothing normal about them.

Three things are worth saying plainly. The theorem says nothing about the shape of the population — it may be skewed, discrete, or bimodal, and \(\overline {X}\) still tends to normality. It does require the variance to be finite; distributions without one are not covered. And the conclusion is about \(\overline {X}\), not about the observations themselves, which keep whatever distribution they always had.

How large \(n\) must be depends on how skewed the population is. For roughly symmetric populations \(n\) of about \(15\) is often ample; the usual rule of thumb, \(n\geq 30\), is meant to cover most of what is met in practice, not to be a threshold at which something switches on.

This density function is a bell-shaped curve that is symmetric about \(\mu \). However, the special normal distribution is the standard normal distribution denoted by \[Z\thicksim N(0,1)\implies \mu =0,\quad \sigma ^2=1.\]

The moment generating function \begin {align*} M_X(t) & = E(e^{tx})\\ & =\int _X e^{tx}f_X(x)\, dx\\ & = \int ^{\infty }_{-\infty } e^{tx}\, \frac {1}{\sigma \sqrt {2\pi }}\, e^{-\frac {1}{2}\left (\frac {x-\mu }{\sigma }\right )^2}\, dx\\ & = \int ^{\infty }_{-\infty } e^{tx}\, \frac {1}{\sigma \sqrt {2\pi }}\, e^{-\frac {1}{2\sigma ^2}\left (x^2 - 2\mu x + \mu ^2\right )}\, dx\\ & = \int ^{\infty }_{-\infty }\frac {1}{\sigma \sqrt {2\pi }}e^{-\frac {1}{2\sigma ^2}\left (x^2 - 2\mu x + \mu ^2 - 2\sigma ^2 tx\right )}\, dx\\ & = \int ^{\infty }_{-\infty }\frac {1}{\sigma \sqrt {2\pi }}\, e^{-\frac {1}{2\sigma ^2}\left (x - \mu - \sigma ^2t\right )^2}\, e^{\mu t + \frac {1}{2}\sigma ^2 t^2}\, dx\\ & = e^{\mu t + \frac {1}{2}\sigma ^2 t^2} \int ^{\infty }_{-\infty }\frac {1}{\sigma \sqrt {2\pi }}\, e^{-\frac {1}{2\sigma ^2}\left (x - \mu - \sigma ^2t\right )^2}\, \, dx\\ & = e^{\mu t + \frac {1}{2}\sigma ^2 t^2}. \end {align*}

Differentiating the moment generating function

\[M_X'(t)=(\mu +t\sigma ^2) e^{\mu t+\frac {1}{2}t^2\sigma ^2}\] \[M''_X(t) = \sigma ^2e^{\mu t+\frac {1}{2}t^2\sigma ^2}+(\mu +t\sigma ^2)(\mu +t\sigma ^2)e^{\mu t +\frac {1}{2}t^2\sigma ^2}\] Then the mean is \[E(x)=M'_X(0)=\mu e^0=\mu \] and \[E(X^2)= M''_X(0) =\sigma ^2e^{0+0}+(\mu +0)(\mu +0)e^{0+0}=\sigma ^2+\mu ^2\] hence, the variance \[\operatorname {Var}(X)=E[x^2]-[E(x)]^2=\sigma ^2+\mu ^2-\mu ^2=\sigma ^2.\]

2.6.6 Cauchy Distribution

In physics, the Cauchy distribution is known as the Lorentz distribution. It is used to model the distribution of energy of unstable states in quantum mechanics and the resonance curves in spectroscopy.

2.6.7 The Weibull Distribution

The distribution is used in reliability and survival analysis to model the lifetime of an object, the lifetime of an organism or systems that fail at a constant rate, or systems that wear out as they age.

2.6.8 Rayleigh Distribution

The Rayleigh distribution is a special case of the Weibull distribution (where the shape parameter is fixed), the only thing that changes here is the scale parameter \(\alpha \).

2.6.9 The Pareto Distribution

The Pareto distribution is a “power-law” distribution traditionally used to model the distribution of wealth and income. In this model, the parameter \(\lambda \) represents the minimum value (such as a minimum wage), while \(\kappa \) determines the shape of the distribution. Beyond economics, it is used to model the lifetime of objects with a mandatory warranty period \(\lambda \), or the duration of events with a guaranteed minimum length \(\lambda \), such as a labor strike.

2.6.10 Practice problems

Show solution

Solution. Let \(T\) be the arrival time in minutes after 10:00, so \(T\sim U(0,30)\) and for \(0\leq t\leq 30\), \(P(T>t) = \frac {30-t}{30}\).

(a). \[P(T>10) = \frac {30-10}{30} = \frac {20}{30} = \frac {2}{3}.\]

(b). Waiting 10 more minutes from 10:15 means \(T\geq 25\), given \(T>15\): \[P(T\geq 25 \mid T>15) = \frac {P(T\geq 25)}{P(T>15)} = \frac {5/30}{15/30} = \frac {5}{15} = \frac {1}{3}.\] Note that this is smaller than the answer to (a). The uniform distribution is not memoryless: having waited already, the remaining wait is bounded by the 30-minute window, so a further 10 minutes becomes less likely, not equally likely. The exponential is the distribution that behaves otherwise, as the next question shows.

Show solution

Solution. Let \(T\sim U(0,30)\) be the passenger’s arrival time in minutes after 7:00. Buses leave at \(0\), \(15\) and \(30\), so the wait is \[W = \begin {cases} 15-T & 0\leq T<15\\ 30-T & 15\leq T<30.\end {cases}\] The wait resets at \(T=15\), so each half of the window has to be looked at separately.

(a). \(W<5\) requires \(15-T<5\) (i.e. \(T>10\)) in the first half, or \(30-T<5\) (i.e. \(T>25\)) in the second. That is \(T\in (10,15)\cup (25,30)\), of total length \(5+5 = 10\): \[P(W<5) = \frac {10}{30} = \frac {1}{3}.\]

(b). \(W>10\) requires \(T<5\) in the first half or \(T<20\) in the second, that is \(T\in [0,5)\cup [15,20)\), again of total length 10: \[P(W>10) = \frac {10}{30} = \frac {1}{3}.\]

Show solution

Solution. For an exponential with mean \(\mu = 100\), the rate is \(\lambda = \tfrac {1}{100}\) and the survival function is \(P(X>x) = e^{-x/100}\).

(a). \(P(X>15) = e^{-0.15} \approx 0.8607\).

(b). \(P(X>110) = e^{-1.1} \approx 0.3329\).

(c). \[P(X>110\mid X>95) = \frac {P(X>110)}{P(X>95)} = \frac {e^{-1.10}}{e^{-0.95}} = e^{-0.15} \approx 0.8607.\] This is exactly the answer to (a), and not by coincidence: the exponential is memoryless. A transistor that has already survived 95 hours is no more likely to fail in the next 15 hours than a brand new one – the component does not age. Contrast the uniform bus in the previous question but one, where waiting made the further wait less likely.

Show solution

Solution. (a). With mean 15, \(P(X>x) = e^{-x/15}\), so \[P(X>17) = e^{-17/15} \approx 0.3220.\]

(b). First, the probability that a single day is below the mean: \[P(X<15) = 1-e^{-15/15} = 1-e^{-1} \approx 0.6321.\] Notice this is not \(\tfrac 12\). The exponential is skewed, so the mean sits above the median – about 63% of days fall below the average rainfall, and a few very wet days pull the mean up.

The five days are independent, each below the mean with probability \(p = 1-e^{-1}\), so the number of such days is binomial: \[P(\text {exactly }4) = \binom {5}{4}p^{4}(1-p) \approx 5(0.6321)^{4}(0.3679) \approx 0.2937.\]

Show solution

Solution. It is quickest to do (b) first and read the others off it. The key is that a gamma density integrates to 1 for any valid shape and scale, so \[\int _0^{\infty }x^{c-1}e^{-x/\beta }\,dx = \beta ^{c}\,\Gamma (c).\]

(b). \[E(X^{k}) = \frac {1}{\beta ^{\alpha }\Gamma (\alpha )}\int _0^{\infty } x^{\alpha +k-1}e^{-x/\beta }\,dx = \frac {\beta ^{\alpha +k}\Gamma (\alpha +k)}{\beta ^{\alpha }\Gamma (\alpha )} = \beta ^{k}\,\frac {\Gamma (\alpha +k)}{\Gamma (\alpha )}.\]

(a). Put \(k=1\) and use \(\Gamma (\alpha +1) = \alpha \Gamma (\alpha )\): \[E(X) = \beta \,\frac {\Gamma (\alpha +1)}{\Gamma (\alpha )} = \alpha \beta .\]

(c). Put \(k=2\), using \(\Gamma (\alpha +2) = (\alpha +1)\alpha \Gamma (\alpha )\): \[E(X^{2}) = \beta ^{2}\alpha (\alpha +1) \implies \operatorname {Var}(X) = \alpha (\alpha +1)\beta ^{2}-(\alpha \beta )^{2} = \alpha \beta ^{2}.\]

Show solution

Solution. Let \(X\) be the number of sixes, so \(X\sim B\left (200,\tfrac 14\right )\) with \[\mu = np = 50, \qquad \sigma ^{2} = npq = 200\left (\tfrac 14\right )\left (\tfrac 34\right ) = 37.5, \qquad \sigma \approx 6.124.\] Since \(np\) and \(nq\) are both comfortably above 5, the normal approximation applies. The word “approximate” in the question is the signal to use it.

Because a discrete count is being approximated by a continuous curve, apply the continuity correction: replace each integer boundary by the half-way point.

(a). “More than 60” means \(X\geq 61\), so use \(60.5\): \[P(X>60) \approx P\left (Z>\frac {60.5-50}{6.124}\right ) = P(Z>1.71) \approx 0.0432.\]

(b). “Less than 45” means \(X\leq 44\), so use \(44.5\): \[P(X<45) \approx P\left (Z<\frac {44.5-50}{6.124}\right ) = P(Z<-0.90) \approx 0.1846.\]

(c). “Between 40 and 55 inclusive” spans \(39.5\) to \(55.5\): \[P(40\leq X\leq 55) \approx P\left (\frac {39.5-50}{6.124}<Z<\frac {55.5-50}{6.124}\right ) = P(-1.71<Z<0.90) \approx 0.7722.\] The exact binomial values are \(0.0454\), \(0.1852\) and \(0.7757\), so the approximation is good to about two decimal places. The correction earns its keep: without it, part (a) would give \(0.0512\) against the true \(0.0454\) – an error more than twice as large as the \(0.0432\) obtained with it.

Show solution

Solution. With mean 3, the rate is \(\lambda = \tfrac 13\) and \(P(X\leq x) = 1-e^{-x/3}\).

(a). \[P(X\leq 5) = 1-e^{-5/3} \approx 1-0.1889 = 0.8111,\] so about 81% of people are affected within 5 minutes.

(b). Solve \(P(X\leq t) = 0.99\) for \(t\): \[1-e^{-t/3} = 0.99 \implies e^{-t/3} = 0.01 \implies -\frac {t}{3} = \ln (0.01),\] \[t = -3\ln (0.01) = 3\ln 100 \approx 13.8 \text { minutes}.\] Note how long the tail is: the average reaction takes 3 minutes, but waiting for 99% certainty takes more than four and a half times that. This is characteristic of the exponential – it is strongly right-skewed, so the mean says little about the extremes.

Show solution

Solution. Variance 25 means \(\sigma = 5\), not 25 – a common slip. So \(X\sim N(68,\,5^{2})\).

(a). Standardise: \[P(X<59) = P\left (Z<\frac {59-68}{5}\right ) = P(Z<-1.8) = 1-\Phi (1.8) = 1-0.9641 = 0.0359.\]

(b). This runs the other way – from a probability back to a score. The top 10% begins at the 90th percentile, and from tables \(\Phi (z) = 0.90\) at \(z \approx 1.28\): \[x = \mu +z\sigma = 68+1.28(5) = 74.4.\] So a student needs about 74.4, in practice a score of 75. Part (a) applies the standardisation forwards and part (b) reverses it; recognising which direction a question runs in is most of the work with normal problems.

Show solution

Solution. The claim is that for an exponential \(X\) with rate \(\lambda \), and any \(s,t>0\), \[P(X>s+t \mid X>s) = P(X>t).\] In words: given that the item has already survived \(s\), the chance it survives a further \(t\) is the same as for a brand new one.

The survival function is \[P(X>x) = \int _x^{\infty }\lambda e^{-\lambda u}\,du = e^{-\lambda x}, \qquad x>0.\] Now, since \(\{X>s+t\}\) is contained in \(\{X>s\}\), their intersection is just \(\{X>s+t\}\), so \[P(X>s+t \mid X>s) = \frac {P(X>s+t)}{P(X>s)} = \frac {e^{-\lambda (s+t)}}{e^{-\lambda s}} = e^{-\lambda t} = P(X>t),\] as required. The whole proof rests on \(e^{-\lambda (s+t)} = e^{-\lambda s}e^{-\lambda t}\): the exponential factorises, so the elapsed time cancels.

The exponential is the only continuous distribution with this property. It is why the exponential models the time to a purely random failure – a component that fails through chance events rather than wear – and why it is the wrong model for something that ages, such as a machine part wearing out or a person’s lifetime.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.