2.1 Introduction
In statistical inference, we assume our data \(X_1, X_2, \dots , X_n\) comes from a distribution \(f(x; \theta )\), but the true value of \(\theta \) is hidden from us. Our goal is to use the sample to construct a “best guess” for the unknown parameter \(\theta \in \Omega \), where \(\Omega \) represents the parameter space.
Definition 2.1.1. A statistic \(T = T(X_1, X_2, \cdots , X_n)\), if a function of the data which does not depend on the unknown parameter \(\theta \).
Example 2.1.2. The sample mean \(\bar {X} = \frac {1}{n}\sum X_i\) is a statistic. While, the quantity \(\frac {\bar {X} - \mu }{\sigma / \sqrt {n}}\) is not a statistic if \(\mu \) or \(\sigma \) are unknown.
Definition 2.1.3. An Estimator is a statistic used to approximate an unknown parameter. It is a random variable denoted by \(\hat {\theta }\).
Definition 2.1.4. An Estimate is the specific numerical value of the estimator obtained from a realized data set \((x_1, x_2, \dots , x_n)\).
Definition 2.1.5. Let \(X_1, X_2, \cdots , X_n\) be a random sample from a population (with a population distribution), then
- (a).
- the \(k^{\text {th}}\) sample moment about 0 is given by \[M_k' = \frac {1}{n}\sum ^n_{i =1}X^k_i\]
- (b).
- the \(k^{\text {th}}\) sample moment about the mean \(\overline {X}\) is given by \[M_k = \frac {1}{n}\sum ^n_{i = 1}(X_i - \overline {X})^k.\]
- 1.
-
- (a)
- \(\mu '_k = E(X^k)\) is the \(k^{\text {th}}\) (population) moment at 0.
- (b)
- \(\mu _k = E(X - \mu )^k\) is the \(k^{\text {th}}\) (population) moment about the mean \(\mu \).
- 2.
- \(M'_1 = \frac {1}{n}\sum ^n_{i = 1}X_i = \overline {X}, \quad M_1 = 0\)
Theorem 2.1.7. Let \(X_1, X_2, \cdots , X_n\) be a random sample from a population (with a population distribution), then \[E(M'_k) = \mu '_k \quad \quad (\text {if}\quad \mu '_k\quad \text {exists})\] \[\operatorname {Var}(M_k') = \frac {1}{n}\left (\mu '_{2k} - (\mu '_k)^2\right ) \quad (\text {if}\quad \mu '_{2k}\quad \text {exist})\]
i.e. if \(k = 1\), then \[E(\overline {X}) = \mu _1' = E(X) = \mu \] \begin {align*} \operatorname {Var}(\overline {X}) & = \frac {1}{n}\left [\mu _2' - (\mu _1')^2 \right ]\\ & = \frac {1}{n}\left [E(X^2) - (E(X))^2\right ]\\ & = \frac {1}{n}\, \operatorname {Var}(X)\\ & = \frac {\sigma ^2}{n}. \end {align*}
Definition 2.1.8. A statistic \(T = T(X_1,X_2, \cdots , X_n)\) that is used to estimate \(\theta \) or \(\tau (\theta )\), a function of \(\theta \), is called an estimator of \(\tau (\theta )\) and an observed value of the statistic \(t = t(x_1, x_2, \cdots , x_n)\) is called an estimate of \(\tau (\theta )\).
Example 2.1.9. If \(X_1, X_2, \cdots , X_n\) are i.i.d random variables with \(E(X_i) = \mu \) then \(\overline {X}\) is an estimator of \(\mu \) and \(\overline {x}\) is an estimate of \(\mu \).
There are a number of methods of finding estimators of \(\theta \) or \(\tau (\theta )\), some of them include:
- 1.
- method of moments (MME)
- 2.
- maximum likelihood method (MLE)
- 3.
- least squares method (mainly in regression) (LSE)
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.