2.1 Introduction
In statistical inference the data \(X_1, X_2, \dots , X_n\) are assumed to come from a distribution \(f(x; \theta )\) whose form is known but whose parameter \(\theta \) is not. The parameter lies in a parameter space \(\Omega \), and the task of this chapter is to use the sample to make a best guess at \(\theta \), or at a function \(\tau (\theta )\) of it, and to say how good the guess is.
Definition 2.1.1. A statistic is a function \(T = T(X_1, X_2, \dots , X_n)\) of the sample that does not depend on any unknown parameter.
Example 2.1.2. The sample mean \(\bar {X} = \frac {1}{n}\sum X_i\) is a statistic. The quantity \(\frac {\bar {X} - \mu }{\sigma / \sqrt {n}}\) is not a statistic if \(\mu \) or \(\sigma \) is unknown, since it cannot be computed from the data alone.
Definition 2.1.3. A statistic \(T = T(X_1,\dots ,X_n)\) used to estimate \(\theta \) or \(\tau (\theta )\) is an estimator; it is a random variable, often written \(\hat \theta \). Its value \(t = T(x_1, \dots , x_n)\) for an observed sample is an estimate.
For example, if \(X_1,\dots ,X_n\) are independent with \(E(X_i) = \mu \), then \(\bar {X}\) is an estimator of \(\mu \) and the number \(\bar {x}\) computed from a particular sample is an estimate of \(\mu \).
Definition 2.1.4. Let \(X_1, X_2, \dots , X_n\) be a random sample. The \(k^{\text {th}}\) sample moment about \(0\) is \(M_k' = \frac {1}{n}\sum ^n_{i =1}X^k_i\), and the \(k^{\text {th}}\) sample moment about the mean is \(M_k = \frac {1}{n}\sum ^n_{i = 1}(X_i - \bar {X})^k\).
The corresponding population moments are \(\mu '_k = E(X^k)\) and \(\mu _k = E\left [(X - \mu )^k\right ]\). Note that \(M'_1 = \bar {X}\) and \(M_1 = 0\).
Theorem 2.1.5. For a random sample, \(E(M'_k) = \mu '_k\) if \(\mu '_k\) exists, and \[\text {Var}(M_k') = \frac {1}{n}\left (\mu '_{2k} - (\mu '_k)^2\right )\] if \(\mu '_{2k}\) exists.
Proof. \(M'_k\) is the mean of the \(n\) independent variables \(X_i^k\), each with mean \(\mu '_k\) and variance \(E(X^{2k}) - [E(X^k)]^2 = \mu '_{2k} - (\mu '_k)^2\). The mean of \(n\) independent identically distributed variables has the same mean and \(\frac 1n\) times the variance. □
With \(k = 1\) this gives \(E(\bar X) = \mu \) and \(\text {Var}(\bar X) = \frac 1n\left [E(X^2) - (E(X))^2\right ] = \frac {\sigma ^2}{n}\).
Several methods produce estimators; the main ones are the method of moments, the method of maximum likelihood, and, mainly in regression, the method of least squares. The rest of the chapter develops these and the criteria for judging the results.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.