1.1 Introduction
Suppose an experiment has been designed and has produced data \(X_1, X_2, \dots , X_n\), where the sample size \(n\) is fixed and known. Statistical inference uses the observed values to learn about the distribution, or the parameter, that generated them. It has two main branches.
- In estimation the observations are used to give an approximate value for some numerical characteristic of the true distribution. The answer may be a single number (point estimation) or a set of plausible values (interval estimation).
- In hypothesis testing the observations are used to decide whether the true distribution, or the true parameter, belongs to a specified subset of the possibilities.
We write the sample compactly as the random vector \(X = (X_1, \dots , X_n)\), and its joint probability (density) function as \(f_\theta (x) = f_\theta (x_1, \dots , x_n)\). A model is a family \(\{f_\theta (x) : \theta \in \Omega \}\), where the parameter space \(\Omega \) is the set of possible values of \(\theta \).
Example 1.1.1. If \(X = (X_1, \dots , X_n)\) is a random sample from \(N(\mu , \sigma ^2)\), then \(\theta = (\mu , \sigma ^2)\), \[f_\theta (x) = \prod ^n_{i = 1} \frac {1}{\sqrt {2\pi \sigma ^2}}\, e^{-\frac {1}{2\sigma ^2}(x_i - \mu )^2} = (2\pi \sigma ^2)^{-n/2}\exp \left (-\frac {1}{2\sigma ^2}\sum _{i=1}^n (x_i - \mu )^2\right ),\] and \(\Omega = \{(\mu , \sigma ^2) : -\infty < \mu < \infty ,\ \sigma ^2 > 0\}\).
Definition 1.1.2. A statistic \(T(X)\) is a function of the data that does not depend on the unknown parameter \(\theta \). An estimator is a statistic used to estimate \(\theta \), or a function \(\tau (\theta )\); its observed value \(T(x)\) is an estimate.
Examples of statistics are the sample mean \(\bar X = \frac 1n\sum X_i\), the sample variance \(S^2 = \frac {1}{n-1}\sum (X_i - \bar X)^2\), the smallest and largest observations \(X_{(1)} = \min _i X_i\) and \(X_{(n)} = \max _i X_i\), and the vector of order statistics \((X_{(1)}, X_{(2)}, \dots , X_{(n)})\). The quantity \(\frac {\bar X - \mu }{\sigma /\sqrt n}\) is not a statistic when \(\mu \) or \(\sigma ^2\) is unknown, since it cannot be computed from the data.
A statistic does not depend on \(\theta \), but its distribution usually does: the distribution of \(\bar X\) moves with \(\mu \). An estimator is therefore a random variable, such as \(\bar X\), while an estimate, such as the number \(\bar x\), is one observed value of it.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.