2.4 Method of Maximum Likelihood
The idea of method of maximum likelihood estimation is to use a value in the parameter space that corresponds to the larges “likelihood” for the observed data as an estimate for the unknown parameter
Definition 2.4.1. Let \(X_1, X_2, \cdots , X_n\) be a random sample from \(f(x;\theta ), \, \theta \in \Omega \) and suppose \(x = (x_1, x_2, \cdots , x_n)\) are observed values of the data. Then \[L(\theta ) = L(\theta ;x) = \prod ^n_{i = 1}f(x_i;\theta ), \quad \theta \in \Omega \] is called the likelihood function of \(\theta \).
Definition 2.4.2. The log likelihood function is defined as \[l(\theta ) = \ln L(\theta ), \quad \theta \in \Omega \] for \(\ln \) the natural logarithmic function.
Definition 2.4.3. The value of \(\theta \) that maximizes the likelihood function \(L(\theta )\) or equivalently the log likelihood function \(l(\theta )\) is called the maximum likelihood (M.L.) estimate. The M.L. estimate is a function of the data \(x = (x_1, x_2, \cdots , x_n)\) and we write \(\hat {\theta } = \hat {\theta }(x_1, x_2, \cdots , x_n)\). The corresponding M.L. estimator is denoted by \(\hat {\theta } = \hat {\theta }(X_1, X_2, \cdots , X_n)\).
Definition 2.4.4. If \(\theta \) is a scalar then the Score function is defined as \[S(\theta ) = S(\theta ;x) = \frac {\partial l(\theta )}{\partial \theta } = \frac {\partial \ln \, L(\theta )}{\partial \theta }\, , \quad \theta \in \Omega \] where \(x = (x_1, x_2, \cdots x_n)\).
If \(\theta = (\theta _1, \theta _2, \cdots , \theta _k)'\) is a vector then the score vector is \[S(\theta ) = S(\theta ; x) = \begin {pmatrix} \frac {\partial l(\theta )}{\partial \theta _1}\\ \frac {\partial l(\theta )}{\partial \theta _2}\\ \vdots \\ \frac {\partial l(\theta )}{\partial \theta _k}\\ \end {pmatrix}\, , \quad \theta \in \Omega .\]
Definition 2.4.5. If \(\theta \) is a scalar then the information function is defined as \[I(\theta ) = I(\theta ;x) = -\frac {\partial ^2 l(\theta )}{\partial \theta ^2} = -\frac {\partial ^2 \ln L(\theta )}{\partial \theta ^2}\, , \quad \theta \in \Omega \] where \(x = (x_1, x_2, \cdots , x_n)\).
If \(\theta = (\theta _1, \theta _2, \cdots , \theta _k)'\) is a vector then the information matrix \(I(\theta )\) is a \(k\times k\) symmetric matrix \[I(\theta ) = \begin {pmatrix} \frac {\partial ^2 l(\theta )}{\partial \theta ^2_1} & \frac {\partial ^2l(\theta )}{\partial \theta _1\partial \theta _2} & \cdots & \frac {\partial ^2 l(\theta )}{\partial \theta _1\partial \theta _k}\\ \frac {\partial ^2 l(\theta )}{\partial \theta _2\partial \theta _1} & \frac {\partial ^2l(\theta )}{\partial \theta ^2_2} & \cdots & \frac {\partial ^2 l(\theta )}{\partial \theta _2\partial \theta _k}\\ \vdots & \vdots & \ddots & \vdots \\ \frac {\partial ^2 l(\theta )}{\partial \theta _k \partial \theta _1} & \frac {\partial ^2 l(\theta )}{\partial \theta _k\partial \theta _2} & \cdots & \frac {\partial ^2l(\theta )}{\partial \theta _k^2}\\ \end {pmatrix}.\]
Note 2.4.6. Although we view the likelihood, log likelihood, score and information functions as functions of \(\theta \), they are also functions of the observed data \(x = (x_1, x_2, \cdots , x_n)\).
Sometimes we write \(L(\theta ;x), \, S(\theta ;x), \, I(\theta ;x)\) to emphasize dependence on the data \(x\).
We also write \(L(\theta ;X), \, S(\theta ;X),\, I(\theta ;X)\) to determine the sampling properties of these functions as functions of the random variable \(X = (X_1, X_2, \cdots , X_n)\).
Finding M.L. Estimates
If \(l(\theta )\) is a twice differentiable function of \(\theta \) on an open interval then we usually find \(\hat {\theta }\) by solving \(S(\theta ) = 0\). If \(\theta \) is a scalar then we can verify that \(\hat {\theta }\) is the M.L. estimate by verifying that \(I(\hat {\theta }) > 0\). Often \(S(\theta ) = 0\) must be solved numerically using an iterative numerical method such as Newton’s Method.
Newton’s Method
Let \(\theta ^{(0)}\) be an initial estimate of \(\theta \), then the estimates are updated by \[\theta ^{(i + 1)} = \theta ^{(i)} + \left [I\left (\theta ^{(i)}\right )\right ]^{-1}S(\theta ^{(i)})\, , \quad i = 0, 1, 2, \cdots \] i.e. \begin {align*} \theta ^{(1)} & = \theta ^{(0)} + \left [I(\theta ^{(0)})\right ]^{-1}\, S(\theta ^{(0)})\\ \theta ^{(2)} & = \theta ^{(1)} + \left [I(\theta ^{(1)})\right ]^{-1}\, S(\theta ^{(1)})\\ \vdots & \end {align*}
until convergence i.e. \(\, \theta ^{(i+ 1)} = \theta ^{(i)}.\)
- 1.
- This algorithm is also called the Newton-Raphson Method.
- 2.
- If the support of \(C\) (domain of \(X\)) depends on \(\theta \) (e.g. UNIF\((0,\theta ))\) then \(\hat {\theta }\) is not found by solving \(S(\theta ) = 0\).
Example 2.4.8. Suppose \(X_1, X_2, \cdots , X_n\) is a random sample from the distribution with p.d.f. \[f(x;\theta ) = \theta x^{\theta - 1}\, , \quad 0\leq x \leq 1, \quad \theta > 0.\] Find the score function, information function and the M.L. estimator of \(\theta \).
Solution. Likelihood function \[L(\theta ) = \prod ^n_{i = 1} f(x;\theta ) = \prod ^n_{i = 1} \theta x_i^{\theta - 1} = \theta ^n\left (\prod ^n_{i = 1} x_i\right )^{\theta - 1}.\] Log likelihood function \begin {align*} l(\theta ) & = \ln L(\theta )\\ & = \ln \left [\theta ^n \left (\prod ^n_{i = 1}x_i\right )^{\theta - 1}\right ]\\ & = \ln \theta ^n + \ln \left (\prod ^n_{i = 1} x_i\right )^{\theta - 1}\\ & = n\ln \theta + (\theta - 1)\ln \prod ^n_{i = 1} x_i\\ & = n\ln \theta + (\theta - 1) \sum ^n_{i = 1} \ln x_i. \end {align*}
Score function \[S(\theta ) = \frac {\partial }{\partial \theta }l(\theta ) = \frac {n}{\theta }+\sum ^n_{i = 1} \ln x_i.\] Information function \[I(\theta ) = -\frac {\partial ^2 }{\partial \theta ^2}l(\theta ) = -\left (-\frac {n}{\theta ^2}\right ) = \frac {n}{\theta ^2}.\] Now, \(S(\theta ) = 0\) \[\frac {n}{\theta } + \sum ^n_{i = 1} \ln x_i = 0\] \[\frac {n}{\theta } = -\sum ^n_{i = 1}\ln x_i\] Therefore, \[\hat {\theta } = -\frac {n}{\sum ^n_{i = 1}\ln X_i}.\] Check: \[I(\hat {\theta }) = \frac {n}{\hat {\theta }^2} = \frac {n}{\left (\frac {-n}{\sum ^n_{i = 1}\ln x_i}\right )^2} = \frac {\left (\sum ^n_{i = 1}\ln X_i\right )^2}{n} > 0\] Therefore, \(I(\hat {\theta }) > 0\)
Hence \[\hat {\theta } = \frac {-n}{\sum ^n_{i = 1}\ln X_i}\] is the M.L. estimator. □
Example 2.4.9. Let \(X_1,X_2,\cdots ,X_n\) be a random sample from uniform on the interval \(\left (\theta -\frac {1}{2},\theta + \frac {1}{2}\right )\). Find the MLE for \(\theta \).
Solution. Likelihood function \[L(\theta ) = \prod ^n_{i=1}f_X(x_i,\theta )=\prod ^n_{i=1}1\cdot I_{(x_i)}\left (\theta -\frac {1}{2},\theta + \frac {1}{2}\right ),\] where \[I_{(x_i)}(a,b)= \begin {cases} 1, & \text {if}\quad x\in (a,b)\\ 0, & \text {if}\quad x\not \in (a,b)\\ \end {cases}.\]
This is known as the indicator function \[L(\theta )=I_{(Y_1)}\left (\theta -\frac {1}{2},\infty \right )\cdot I_{(Y_n)}\left (-\infty ,\theta +\frac {1}{2}\right ).\] \(Y_1=\min \{X_1,X_2,\cdots ,X_n\}\) and \(Y_n=\max \{X_1,X_2,\cdots ,X_n\}\). The value of \(\theta \) to maximize \(L(\theta )\) must be such that \(Y_1>\theta -\frac {1}{2}\) and \(Y_n<\theta +\frac {1}{2}\). \[\theta < Y_1+\frac {1}{2}\hspace {0.5cm}\text {and}\hspace {0.5cm} \theta >Y_n-\frac {1}{2}\]
\[\implies \hspace {0.5cm} Y_n-\frac {1}{2}<\theta <Y_1+\frac {1}{2}\]
Any value of \(\theta \) in \(\left (Y_n-\frac {1}{2},Y_1+\frac {1}{2}\right )\) in MLE for \(\theta \). We usually take the average of the two end points \[\hat {\theta }_{\text {MLE}}=\frac {Y_n+Y_1}{2}.\] □
Example 2.4.10. Suppose \(X_1, X_2, \cdots , X_n\) is a random sample from the p.d.f \[f(x;\beta ) = \beta x^{\beta - 1} e^{-x^{\beta }}\, , \quad x\geq 0, \quad \beta > 0.\] Find the M.L. estimator for \(\beta \).
Solution. Likelihood function \[L(\beta ) = \prod ^n_{i = 1}\beta x_i^{\beta - 1} e^{-x_i^{\beta }} = \beta ^n \left (\prod ^n_{i = 1}x_i\right )^{\beta - 1}\, e^{-\sum ^n_{i = 1}x_i^{\beta }}.\] Log likelihood function \begin {align*} l(\beta ) & = \ln L(\beta )\\ & = n\ln \beta + (\beta - 1)\ln \prod ^n_{i = 1}x_i - \sum ^n_{i = 1} x_i^{\beta }\\ & = n\ln \beta + (\beta - 1)\sum ^n_{i = 1}\ln x_i - \sum ^n_{i = 1}x_i^{\beta }. \end {align*}
Score function \[S(\beta ) = \frac {\partial }{\partial \beta }l(\beta ) = \frac {n}{\beta }+ \sum ^n_{i = 1}\ln x_i - \sum ^n_{i = 1} x_i^{\beta } \ln x_i.\] \(S(\beta ) = 0\) cannot be solved explicitly, so use Newton’s method \begin {align*} I(\beta ) & = -\frac {\partial ^2l(\beta )}{\partial \beta ^2}\\ & = -\left [-\frac {n}{\beta ^2}- \sum ^n_{i = 1} x_i^{\beta } (\ln x_i)^2\right ]\\ & = \frac {n}{\beta ^2} + \sum ^n_{i = 1} x_i^{\beta }(\ln x_i)^2. \end {align*}
Let \(\, \beta ^{(0)} =\) initial guess. \[\beta ^{(i + 1)} = \beta ^{(i)} + \left [I(\beta ^{(i)})\right ]^{-1}S(\beta ^{(i)}), \quad i = 0, 1, 2, \cdots \] iterate until convergence. □
Theorem 2.4.11 (Invariance property). Suppose \(\theta \) is a scalar and \(\tau = h(\theta )\) is a one-to-one function of \(\theta \). If \(\hat {\theta }\) is the M.L. estimator of \(\theta \), then \(\hat {\tau } = h(\hat {\theta })\) is the M.L. estimator of \(\tau \).
Example 2.4.12. If \(X_1, X_2, \cdots , X_n\) is a random sample from \[f(x;\theta ) = \frac {\theta ^x e^{-\theta }}{x!}\, , \quad x = 0, 1, 2, \cdots \] Find the M.L. estimator of \(e^{-\frac {1}{\theta }}\).
Solution. Likelihood function \[L(\theta ) = \prod ^n_{i = 1}f(x_i;\theta ) = \prod ^n_{i = 1}\frac {\theta ^{x_i}e^{-\theta }}{x_i!} = \frac {\theta ^{\sum ^n_{i = 1}x_i}e^{-n\theta }}{\prod ^n_{i = 1}x_i!}.\] Log likelihood function \begin {align*} l(\theta ) & = \ln L(\theta )\\ & = \ln \theta ^{\sum _{i = 1}^nx_i} + \ln e^{-n\theta } - \ln \prod ^n_{i = 1} x_i!\\ & = \sum ^n_{i = 1} x_i \, \ln \theta - n\theta - \sum ^n_{i = 1} \ln x_i. \end {align*}
Score function \[S(\theta ) = \frac {\partial }{\partial \theta }l(\theta ) = \frac {1}{\theta }\sum ^n_{i = 1} x_i - n.\] \(S(\theta ) = 0\) \[\frac {1}{\theta }\sum ^n_{i = 1} x_i = n\] \[\hat {\theta } = \frac {1}{n}\sum ^n_{i = 1} X_i = \overline {X}.\] i.e. \(\hat {\theta } = \overline {X}\) is the M.L. estimator for \(\theta \).
Therefore, \(\hat {\tau } = e^{-\frac {1}{\overline {X}}}\) is the M.L. estimator for \(e^{-\frac {1}{\theta }}\). □
Definition 2.4.13. If \(\theta \) is a scalar then the Fisher information (function) is given by \[J(\theta ) = E\left [I(\theta ;X)\right ] = E\left [-\frac {\partial ^2 l(\theta ;X)}{\partial \theta ^2}\right ]\] where \(X = (X_1, X_2, \cdots ,X_n)\).
If \(\theta = (\theta _1, \theta _2, \cdots , \theta _k)'\) is a vector then the Fisher information matrix \(J(\theta )\) is a \(k\times k\) symmetric matrix whose \((i,j)^{\text {th}}\) entry is given by \[E\left [-\frac {\partial ^2 l(\theta ; X)}{\partial \theta _i\partial \theta _j}\right ]\] where \(X = (X_1, X_2, \cdots X_n)\).
Note 2.4.14. If \(X_1, X_2, \cdots , X_n\) is a random sample from \(f(x;\theta )\) then \[E\left [\frac {-\partial ^2l(\theta ;X_1, \cdots , X_n)}{\partial \theta _i \, \partial \theta _j}\right ] = nE\left [\frac {-\partial ^2 \ln f(X;\theta )}{\partial \theta _i\, \partial \theta _j}\right ]\] where \(X\) has PDF \(f(x;\theta )\).
Example 2.4.15. Let \(X_1, X_2, \cdots , X_n\) be a random sample from \[f(x;\theta ) = \frac {1}{\theta }e^{-\frac {x}{\theta }}\, , \quad x > 0.\] Find the score function, the information function, the Fisher information function and the M.L. estimator of \(\theta \).
Solution. Likelihood function \[L(\theta ) = \prod ^n_{i =1}f(x_i;\theta ) = \prod ^n_{i = 1} \frac {1}{\theta }e^{-\frac {x_i}{\theta }}= \frac {1}{\theta ^n}e^{-\frac {1}{\theta }\sum ^n_{i = 1} x_i}\]
The log likelihood function \[l(\theta ) = \ln L(\theta ) = -\frac {1}{\theta }\sum ^n_{i = 1} x_i - n\ln \theta .\] Score function \[S(\theta ) = \frac {\partial l(\theta )}{\partial \theta }=\frac {1}{\theta ^2}\sum ^n_{i = 1} x_i - \frac {n}{\theta }.\] Information function \[I(\theta ) = -\frac {\partial ^2l(\theta )}{\partial \theta ^2}=-\left [-\frac {2}{\theta ^3}\sum ^n_{i = 1} x_i + \frac {n}{\theta ^2}\right ] = \frac {2}{\theta ^3}\sum _{i = 1}^n x_i - \frac {n}{\theta ^2}.\] \(E(X_i)\) \begin {align*} E(X_i) & = \int ^{\infty }_0 x_i \, \frac {1}{\theta }e^{-\frac {x_i}{\theta }}\, dx_i\\ & = \int ^{\infty }_0 \, \frac {x_i}{\theta }e^{-\frac {x_i}{\theta }}\, dx_i\\ & = \theta \underbrace {\int ^{\infty }_0m_i e^{-m_i}\, dm_i}_{1!}\\ & = \theta . \end {align*}
Fisher information \begin {align*} J(\theta ) & = E[I(\theta ;X)]\\ & = E\left [\frac {2}{\theta ^3}\sum ^n_{i = 1}X_i - \frac {n}{\theta ^2}\right ]\\ & = \frac {2}{\theta ^3}\sum ^n_{i = 1}E(X_i) - \frac {n}{\theta ^2}\\ & = \frac {2}{\theta ^3}\sum ^n_{i = 1} \theta - \frac {n}{\theta ^2}\\ & = \frac {2n\theta }{\theta ^3}-\frac {n}{\theta ^2}\\ & = \frac {2n}{\theta ^2}-\frac {n}{\theta ^2}\\ & = \frac {n}{\theta ^2}. \end {align*}
Solving \(S(\theta ) = 0\) \[\frac {1}{\theta ^2}\sum ^n_{i = 1} x_i = \frac {n}{\theta }\] \[\frac {1}{n}\sum ^n_{i = 1} = \theta \] Therefore, \(\widehat {\theta } = \frac {1}{n}\sum ^n_{i = 1} X_i\). □
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.