2.1 The Four Distributions You Will Need

Everything that follows – confidence intervals, hypothesis tests, analysis of variance, regression – is built from four distributions. They are introduced together here because the hardest part is not the arithmetic but knowing which one applies, and that is easier to see when they are side by side.

2.1.1 The normal distribution

A continuous variable \(X\) with mean \(\mu \) and variance \(\sigma ^2\) is normal, written \(X\sim N(\mu ,\sigma ^2)\), if its curve is the familiar symmetric bell centred at \(\mu \). Its properties:

1.

It is symmetric about \(\mu \), so mean, median and mode coincide there.

2.

The total area under the curve is 1, and \(\sigma \) controls the spread – a larger \(\sigma \) gives a flatter, wider curve.

3.

Probabilities are areas: \(P(a<X<b)\) is the area under the curve between \(a\) and \(b\).

There is a different normal curve for every pair \((\mu ,\sigma )\), so tables for each are out of the question. Instead we standardise: \[Z=\frac {X-\mu }{\sigma } \sim N(0,1).\] \(Z\) counts how many standard deviations a value sits from its mean, which is a pure number with no units. The standard normal \(N(0,1)\) is centred at 0 with \(P(Z<0)=P(Z>0)=0.5\), and it is the one distribution that needs tabulating.

The notation \(Z_{\alpha }\) means the value with area \(\alpha \) to its right, so \(P(Z>Z_{\alpha })=\alpha \). The ones worth knowing by heart are \[Z_{0.05}=1.645, \qquad Z_{0.025}=1.96, \qquad Z_{0.005}=2.576,\] which give the \(90\%\), \(95\%\) and \(99\%\) two-sided intervals. Because the curve is symmetric, \(Z_{1-\alpha }=-Z_{\alpha }\) – one table serves both tails.

2.1.2 Why the sample mean is normal: the Central Limit Theorem

The reason the normal distribution dominates a course about samples is that \(\overline {X}\) tends to be normal even when the individual observations are not.

The word approximately is doing real work: \(Z\) is not exactly \(N(0,1)\) for any finite \(n\), it only gets closer as \(n\) grows. In practice \(n\geq 30\) is the usual rule of thumb, and like all such rules it is a convenience rather than a threshold — a badly skewed population needs more, a roughly symmetric one needs less.

Two consequences worth noticing. The spread of \(\overline {X}\) is \(\sigma /\sqrt {n}\), not \(\sigma \) – averages vary less than individual observations, which is why sampling works at all. And the \(\sqrt {n}\) means precision improves slowly: to halve the standard error you must quadruple the sample.

2.1.3 The \(t\)-distribution

The formula above needs \(\sigma \), which is almost never known. Replacing it by the sample value \(\hat {S}\) introduces a second source of variation, and the result is no longer normal: \[t=\frac {\overline {X}-\mu }{\hat {S}/\sqrt {n}} \sim t_{n-1}.\] Its properties:

1.

It is symmetric about 0, like the standard normal, but with heavier tails – wider, to pay for the extra uncertainty in estimating \(\sigma \).

2.

It has one parameter, the degrees of freedom, here \(n-1\). One is lost because \(\overline {X}\) had to be estimated before \(\hat {S}\) could be computed.

3.

As the degrees of freedom grow, \(t\) approaches \(N(0,1)\). By about \(n=30\) the difference hardly matters, which is why large-sample work uses \(Z\).

So \(t_{n-1,\alpha }\) is always a little larger than \(Z_{\alpha }\), and confidence intervals built with \(t\) are correspondingly wider. That is not a defect: it is the price of not knowing \(\sigma \).

2.1.4 The chi-square distribution

Where the normal and \(t\) concern means, the chi-square distribution, \(\chi ^2_{v}\), concerns squared quantities – variances and counts.

1.

It takes only non-negative values, since it is built from squares, and it is not symmetric. It is skewed to the right, becoming more symmetric as the degrees of freedom increase.

2.

Because it is not symmetric, the two tails must be looked up separately: \(\chi ^2_{1-\alpha /2}\) and \(\chi ^2_{\alpha /2}\) are not negatives of one another, and there is no shortcut of the kind symmetry allows for \(Z\) and \(t\).

It appears twice in this course: for inference about a variance, where \((n-1)\hat {S}^2/\sigma ^2\sim \chi ^2_{n-1}\), and in the goodness-of-fit and independence tests, where the statistic sums squared discrepancies between observed and expected counts.

2.1.5 The \(F\)-distribution

The \(F\)-distribution arises when two variances are compared, as a ratio: \[F=\frac {\hat {S}_1^2/\sigma _1^2}{\hat {S}_2^2/\sigma _2^2} \sim F_{v_1,\,v_2}.\]

1.

It has two degrees-of-freedom parameters – numerator \(v_1\) and denominator \(v_2\) – and the order matters: \(F_{5,20}\) is not \(F_{20,5}\).

2.

It is positive and right-skewed, being a ratio of non-negative quantities.

3.

Tables usually give only the upper tail. The lower tail is recovered from \[F_{1-\alpha ,\,v_1,v_2}=\frac {1}{F_{\alpha ,\,v_2,v_1}},\] noting that the degrees of freedom swap.

It is used for testing the equality of two variances, and – as the next chapter shows – throughout the analysis of variance, where the whole method rests on comparing variation between groups with variation within them.

2.1.6 Choosing between them

\begin {align*} \textbf {Notation:} & \\ \overline {X}&=\hspace {0.3cm}\text {Sample Mean}\\ S^2&=\hspace {0.3cm}\text {Sample Variance}\\ S&=\hspace {0.3cm}\text {Sample s.d.}\\ \mu &=\hspace {0.3cm}\text {Population Mean}\\ \sigma ^2&=\hspace {0.3cm}\text {Population Variance}\\ \sigma &=\hspace {0.3cm}\text {Population s.d.}\\ \end {align*}

We have the theory of probability and now we want the objective of statistics.

Statistics is about making inference about a population based on information contained in a sample.
Generally, population are characterized by numerical descriptive measures and the objective of many investigation is to make an inference about one or more population parameters

Methods of making inference about population parameters falls in one of two categories:

1.

estimation of parameters.

2.

decision making concerning the value of the parameter.

Methods of estimation can be classified into:

1.

point estimation

2.

interval estimation

A point estimation procedure utilize information in a sample to arrive at s single number or point which estimates the target population.

2.1.7 Properties of Estimation

Suppose we want to estimate a population parameter \(\theta \). The estimation of \(\theta \) will be indicated by the symbol \(\hat {\theta }\). We would like the distribution of estimates of the probability distribution of the estimation to center about the target parameters.

Probability distribution for a positively biased estimator.

\begin {equation} \label {eq:var-xbar} \operatorname {var}(\overline {X})=\frac {\sigma ^2}{n},\qquad \text {s.e}(\overline {X})=\frac {\sigma }{\sqrt {n}} \end {equation}

Independence is the step that needs the ”with replacement”. Without it the draws are not independent and the variances do not simply add.

2.1.8 Estimating the population variance

Two candidates suggest themselves for estimating \(\sigma ^2\): \[\frac {\sum (X-\mu )^2}{n}\hspace {1cm}\text {(unbiased, but needs }\mu \text {)}\] \[S^2=\frac {\sum (X-\overline {X})^2}{n}\hspace {1cm}\text {(computable, but biased)}\] The first is unaffected by the problem below, but it requires \(\mu \), which is exactly what is unknown. The second uses \(\overline {X}\) in its place — and that substitution costs something.

So \(\hat {S}^2\) is the unbiased estimator, and the two are related by \[\hat {S}^2=\frac {n}{n-1}\,S^2.\] Note what is and is not being claimed: \(E(\hat {S}^2)=\sigma ^2\), not \(\hat {S}^2=\sigma ^2\). The estimator is right on average over all possible samples; any particular sample will give something near \(\sigma ^2\), not equal to it.

2.1.9 An Unbiased Estimation of Population Proportion

Suppose each member of a population either has some attribute or does not, and the proportion having it is \(P\). In a sample of \(n\), the number \(X\) that have it is binomial, \(X\sim B(n,P)\), with \(E(X)=nP\) and \(\operatorname {var}(X)=nP(1-P)\). The natural estimator of \(P\) is the sample proportion \[P_s=\frac {X}{n}.\]

For large \(n\) the binomial distribution of \(X\) is approximately normal, and therefore so is \(P_s\): \[P_s\ \text {is approximately}\ N\!\left (P,\ \frac {P(1-P)}{n}\right ),\qquad \text {so}\qquad Z=\frac {P_s-P}{\sqrt {P(1-P)/n}}\ \text {is approximately}\ N(0,1).\] It is the standardised quantity that is \(N(0,1)\), not \(P_s\) itself, and the mean of \(P_s\) is \(P\) — not the estimate \(\frac {X}{n}\) obtained from one particular sample.

2.1.10 Expected Values And Variances of Some Common Point Estimators

The probability distribution of an estimator \(\hat {\theta }\), whether we know the probability distributes of \(\hat {\theta }\) or not; an approximated bound on \(\epsilon \) for unbiased estimators can be found by expressing \(b\) as a multiple s.d. of \(\hat {\theta }\) say \(b=k\sqrt {\hat {\theta }}\) and the default for \(k=2\), \(b=2\sqrt {\hat {\theta }}\).

2.1.11 Confidence Interval

An interval estimate gives a range of values which has a certain probability of containing the population parameter. We find the interval estimates for population mean \(\mu \) where \(\sigma =\) unknown or \(\sigma =\) unknowing.
Also we consider the sample size \(n\geq 30\) is called large sample size and we use \(Z\) - table.
We can use this distribution to find a range of the value for which \(\overline {X}\) will lie within a certain probability.

\(\alpha =\) proportion of missing the correct value. \(\alpha \) level of significance.

Our confidence interval is given by: \(\overline {X}\pm Z_{\alpha /2}\times \text {SE}\)

\begin {align*} \overline {X} &=\hspace {0.3cm} \text {Sample mean}\\ Z_{\alpha /2} &=\hspace {0.3cm}\text {degree of confidence table value with given}\hspace {0.2cm}\alpha .\\ \text {SE} &=\hspace {0.3cm}\text {Standard error}\\ \end {align*}

2.1.12 Properties of Confidence Intervals

Which multiplier to use, \(Z\) or \(t\), is decided by one question only: is \(\sigma \) known? It is not decided by the sample size. The \(t\)-distribution exists because replacing \(\sigma \) by an estimate \(\hat {S}\) adds a second source of uncertainty, and \(t\) is wider than \(Z\) to pay for it. If \(\sigma \) is genuinely known there is nothing extra to pay for, so \(Z\) is correct however small the sample.

Sample size decides something different: whether the interval is trustworthy at all. For small \(n\) these formulas need the population to be roughly normal; for large \(n\) the Central Limit Theorem makes that assumption unnecessary.

1.

\(\sigma \) known, any \(n\) (population normal, or \(n\) large): \[\overline {X}\pm Z_{\alpha /2}\times \frac {\sigma }{\sqrt {n}}\]

2.

\(\sigma \) unknown, \(n\) large: \[\overline {X}\pm Z_{\alpha /2}\times \frac {\hat {S}}{\sqrt {n}} \hspace {0.5cm}\text {equivalently}\hspace {0.5cm} \overline {X}\pm Z_{\alpha /2}\times \frac {S}{\sqrt {n-1}}\] The two forms agree because \(S\) is computed with divisor \(n\) and \(\hat {S}\) with divisor \(n-1\), so \(\hat {S}/\sqrt {n}=S/\sqrt {n-1}\). Strictly \(t\) is correct here too, but for large \(n\) the two are indistinguishable.

3.

\(\sigma \) unknown, \(n\) small (population normal): \[\overline {X}\pm t_{n-1,\,\alpha /2}\times \frac {\hat {S}}{\sqrt {n}} \hspace {0.5cm}\text {equivalently}\hspace {0.5cm} \overline {X}\pm t_{n-1,\,\alpha /2}\times \frac {S}{\sqrt {n-1}}\] This is the only case that calls for \(t\).

4.

C.I. for a proportion, \(n\) large enough that \(n\hat {P}\geq 5\) and \(n(1-\hat {P})\geq 5\): \[\hat {P}\pm Z_{\alpha /2}\times \sqrt {\frac {\hat {P}(1-\hat {P})}{n}}\] Note it is the sample proportion \(\hat {P}\) throughout: if \(P\) were known there would be nothing to estimate. When the sample is too small for that condition, \(t\) does not rescue it – a proportion is a count from a binomial, and there is no estimated variance for \(t\) to correct. The proper remedies are the Wilson score interval or the exact Clopper–Pearson interval, both beyond this course.

5.

C.I. for the difference between two means, \(\sigma _1\) and \(\sigma _2\) known: \[(\overline {X}_1-\overline {X}_2)\pm Z_{\alpha /2}\times \sqrt {\frac {\sigma _1^2}{n_1}+\frac {\sigma _2^2}{n_2}}\]

6.

C.I. for the difference between two proportions: \[(\hat {P}_1-\hat {P}_2)\pm Z_{\alpha /2}\times \sqrt {\frac {\hat {P}_1(1-\hat {P}_1)}{n_1}+\frac {\hat {P}_2(1-\hat {P}_2)}{n_2}}\]

7.

C.I. for the difference between two means when \(\sigma _1=\sigma _2\) but both are unknown. The two samples are pooled into a single estimate of the common standard deviation: \[(\overline {X}_1-\overline {X}_2)\pm t_{n_1+n_2-2,\,\alpha /2}\times S_p\sqrt {\frac {1}{n_1}+\frac {1}{n_2}} \hspace {0.4cm}\text {where}\hspace {0.4cm} S_p=\sqrt {\frac {(n_1-1)\hat {S}^2_1+(n_2-1)\hat {S}^2_2}{n_1+n_2-2}}\] The multiplier is \(t\), not \(Z\), because \(\sigma \) is unknown – the same rule as case 3 – and the degrees of freedom are \(n_1+n_2-2\), one lost from each sample. Pooling is only reasonable if the two variances really are close; a common rough check is \[\frac {1}{2}<\frac {\hat {S}^2_1}{\hat {S}^2_2}<2.\]

\(\bullet \)

The number we add to and subtract from the point estimate is called the “margin of error”. \[\text {Point estimate}\, \pm \, \text {Margin error}\] confidence level is denoted by \(\hspace {0.2cm}(1-\alpha )\, 100\%\)

\(\bullet \)

Margin of error for the estimate for \(\mu \) denoted by \(E\), is the quantity that is subtracted from and added to the value of \(\overline {X}\) to obtain a confidence interval for \(\mu \). Thus \[E = Z\,\cdot \,\sigma _{\overline {X}} = Z\,\cdot \, \frac {\sigma }{n}\]

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.