3.4 Bayes Estimation
There are two major schools of thought in statistical inference, the frequentist and Bayesian
School. Frequentist consider a parameter as an unknown constant underlying a probability
(density) function of the data.
Bayesians consider the parameter \(\theta \) to be random with a prior distribution \(\pi (\theta )\). The purpose of the
data then is to adjust this distribution for \(\theta \), to result in a posterior distribution of the
parameter.
Theorem 3.4.1 (Posterior Distribution). Suppose a parameter is chosed at random according to the prior distribution \(\pi (\theta )\), then the posterior distribution of the parameter \(\theta \) given the data \(x = (x_1, \, \cdots \, , \, x_n)\) of the parameter is the conditional distribution of \(\theta \) given the data \(x = (x_1, \, \cdots \, , \, x_n)\) \[\pi (\theta /x) = C\, \pi (\theta )\, \prod ^n_{i = 1} f_{\theta }(x_i) = C\, \pi (\theta )\, L(\theta )\] where \[C = \frac {1}{\int ^{\infty }_{-\infty } \pi (\theta )\, L(\theta )\, d\theta }\] and \(L(\theta )\) is the likelihood function.
Example 3.4.2. Suppose a coin is tossed \(n\) times with probability of heads \(\theta \). Assume that the prior probability of heads follows a \(BETA(10,10)\) distribution. If the \(n\) tosses result in \(x\) heads, find the posterior density function of \(\theta \).
Solution. \(f_{\theta }(x) = \binom {n}{x}\,\theta ^x\,(1 - \theta )^{n - x}\) \[L(\theta ) = \binom {n}{x}\, \theta ^x\, (1 - \theta )^{n - x}\] \[\theta \thicksim BETA(10,10)\] \[\pi (\theta ) = \frac {1}{B(10,10)} \, \theta ^{10 - 1}\, (1 - \theta )^{10 - 1} = C_1\, \theta ^9 \, (1 - \theta )^{9}\, , \hspace {0.3cm} 0 < \theta < 1.\] \begin {align*} \pi (\theta / x) & = C_2\, \pi (\theta )\, L(\theta )\\ & = C_3 \, \theta ^9\, (1 - \theta )^{9}\, \theta ^x\, (1 - \theta )^{n - x}\\ & = C_3\, \theta ^{x + 9}\, (1 - \theta )^{n + 9 - x}. \end {align*}
\[C_3 = \frac {1}{\int _0^1\theta ^{x + 9}\, (1 - \theta )^{n + 9 - x}\, d\theta } = \frac {1}{B(x + 10, n + 10 - x)}.\]
\[\pi (\theta /x) = \frac {\theta ^{x + 9} (1 - \theta )^{n + 9 - x}}{B(x + 10, n + 10 -x)}\] \[\theta / x \, \thicksim \, BETA(10 + x, 10 + n - x)\] □
Recall: \[B(\alpha ,\beta ) = \frac {\Gamma (\alpha )\, \Gamma (\beta )}{\Gamma (\alpha + \beta )} = \int ^1_0x^{\alpha - 1}\, (1 - x)^{\beta - 1}\, dx.\]
Theorem 3.4.3 (Conjugate Prior Distribution). If a prior distribution has the property that the posterior distribution is in the same family of distributions as the prior then the prior is called a conjugate prior.
Example 3.4.4. Let \(x_1, \, X_2, \, \cdots \, , \, x_n\) be a random sample from the distribution with pdf \[f_{\theta }(x) = \theta \, x^{\theta - 1}\, , \hspace {0.3cm} 0 < x < 1, \, \hspace {0.2cm} \theta > 1.\] Show that the prior distribution \(\theta \thicksim GAM(a + 1\, , \, 1/b)\) is a conjugate prior.
Solution. \begin {align*} \pi (\theta ) & = \frac {1}{\left (\frac {1}{b}\right )^{a + 1}\, \Gamma (a + 1)} \, \theta ^a\, e^{-\theta /(\frac {1}{b})}\hspace {0.2cm}, \hspace {0.3cm} \theta > 0\\\\ & = K\, \theta ^a\, e^{-b\theta }\, \, , \, \hspace {0.3cm} \theta > 0. \end {align*}
\begin {align*} L(\theta ) & = \prod ^n_{i = 1} \theta \, x_i^{\theta } = \theta ^n\, \left (\prod ^n_{i = 1} x_i \right )^{\theta - 1}\\ & = \theta ^n\, \left (\prod ^n_{i = 1} x_i\right )^{\theta - 1}\, \frac {1}{\prod ^n_{i = 1} x_i}\\ & = K_1\, \theta ^n\, \left (\prod ^n_{i = 1} x_i \right )^{\theta }. \end {align*}
\begin {align*} \pi (\theta /x) & = C\, \pi (\theta )\, L(\theta )\\ & = C_1\, \theta ^a\, e^{-b\theta }\, \theta ^n\, \left (\prod ^n_{i = 1} x_i\right )^{\theta }\\ & = C_1\, \theta ^{a + n}\, e^{-b\theta }\, e^{\log \left (\prod ^n_{i = 1} x_i\right )^{\theta }}\\ & = C_1\, \theta ^{a + n}\, e^{-b\theta }\, e^{\theta \, \log \prod ^n_{i = 1} x_i}\\ & = C_1\, \theta ^{a + n} \, e^{\theta \left (b - \log \prod ^n_{i = 1} x_i\right )}\\ & = C_1\, \theta ^{a + n}\, e^{-\theta /\frac {1}{b - \sum ^n_{i = 1} \log x_i}}. \end {align*}
\[\therefore \,\, \theta \mid x\, \sim \, GAM\left (a + n + 1\, , \, \frac {1}{b - \sum ^n_{i = 1} \log x_i}\right ).\]
\(\therefore \, \theta \, \thicksim \, GAM(a + 1, 1/b)\) is a conjugate prior since \(\theta \mid x\) is also gamma. □
Bayes point estimation
One method of obtaining a point estimator of \(\theta \) is to use the posterior distribution together with a suitable loss function. Which estimator results depends on the loss chosen, and the next two results give the two standard cases.
Theorem 3.4.5. The Bayes estimator of \(\theta \) for a squared error loss function with respect to the prior \(\pi (\theta )\) given data \(X\) is the mean of the posterior distribution given by \[\tilde {\theta } = \tilde {\theta }(X) = \int _{-\infty }^{\infty } \theta \, \pi \left (\theta \mid x\right )\, d\theta .\] This estimator minimizes \[E\left [(\tilde {\theta } - \theta )^2| X\right ] = \int _{-\infty }^{\infty } \left \{\int _{-\infty }^{\infty } (\title {\theta } -\theta )^2\, f_{\theta }(x)\, dx\right \}\, \pi (\theta )\, d\theta \]
Example 3.4.6. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the distribution with pdf \[f_{\theta }(x) = \theta \, x^{\theta - 1}\, , \, \hspace {0.2cm} 0 < x < 1\,, \, \hspace {0.2cm} \theta > 1.\] Using the prior distribution \(\theta \thicksim GAM\left (a + 1, 1/b\right )\), find the Bayes estimator of \(\theta \) for the squared error loss function.
Solution. From Example 3.4.4 \[\pi (\theta / x) = C_1\, \theta ^{a + n}\, e^{-\theta /\left (\frac {1}{b - \sum ^n_{i = 1} \log X_i}\right )}\] \[\theta /X \, \thicksim \, GAM\left (a+n + 1\, , \,\frac {1}{b - \sum ^n_{i = 1} \log X_i}\right )\] \[\tilde {\theta } = E\left [\theta /X\right ] =\frac {a + n + 1}{b - \sum ^n_{i = 1}\log X_i}.\] □
Problem 3.4.1. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(POI(\theta )\) distribution.
- (a)
- Show that the prior distribution \[\theta \, \thicksim \, GAM\left (b + 1, \frac {1}{a}\right )\] is a conjugate prior.
- (b)
- using the conjugate prior in (a) find the Bayes estimator of \(\theta \) for a squared error loss function.
Show solution
Solution.
(a) Conjugacy
Write \(T=\sum _{i=1}^{n}X_i\). The likelihood and the GAM\(\left (b+1,1/a\right )\) prior are \[L(\theta ) \propto e^{-n\theta }\theta ^{T}, \qquad \pi (\theta ) \propto \theta ^{b}e^{-a\theta },\quad \theta >0 .\] Multiplying, \[\pi \left (\theta \mid x\right ) \propto \theta ^{\,T+b}\,e^{-(n+a)\theta },\] which is the kernel of a GAM\(\left (T+b+1,\ \dfrac {1}{n+a}\right )\) density. The posterior belongs to the same gamma family as the prior, so the family is conjugate for the Poisson likelihood.
(b) The Bayes estimator
Under squared error loss the Bayes estimator is the posterior mean. A GAM\((\kappa ,\lambda )\) variable has mean \(\kappa \lambda \), so \[\widetilde {\theta } = \frac {T+b+1}{n+a} = \frac {\sum _{i=1}^{n}X_i + b + 1}{n+a} .\]
Note. Write it as a weighted average: \[\widetilde {\theta } = \frac {n}{n+a}\,\overline {X} + \frac {a}{n+a}\cdot \frac {b+1}{a},\] a compromise between the sample mean and the prior mean \((b+1)/a\), with weights \(n\) and \(a\). So \(a\) acts as a prior sample size: the prior carries as much influence as \(a\) observations, and its influence vanishes as \(n\) grows.
Problem 3.4.2. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(X\thicksim N(0, \sigma ^2)\) distribution where \(\sigma ^2\) is known. Assuming the prior distribution is \(\theta \, \thicksim \, N\left (\frac {b}{a}, \frac {\sigma ^2}{a}\right ),\) fined the Bayes estimator using a squared error loss function.
Show solution
Solution. The model must be \(X_i \sim N\left (\theta ,\sigma ^{2}\right )\) with \(\sigma ^{2}\) known — as printed the mean is zero and \(\theta \) does not appear in the likelihood at all, so there would be nothing to estimate.
With \(T=\sum _i X_i\) and prior \(\theta \sim N\left (b/a,\ \sigma ^{2}/a\right )\), \[\pi \left (\theta \mid x\right ) \propto \exp \left \{-\frac {1}{2\sigma ^{2}}\sum _i\left (x_i-\theta \right )^{2}\right \} \exp \left \{-\frac {a}{2\sigma ^{2}}\left (\theta -\frac {b}{a}\right )^{2}\right \}.\] Both exponents are quadratic in \(\theta \), so the posterior is normal and it is enough to collect the coefficients. Discarding terms free of \(\theta \), the exponent is \[-\frac {1}{2\sigma ^{2}}\left [(n+a)\theta ^{2} - 2\theta \left (T+b\right )\right ] = -\frac {n+a}{2\sigma ^{2}}\left [\theta - \frac {T+b}{n+a}\right ]^{2} + \text {const} .\] Hence \[\theta \mid x \ \sim \ N\!\left (\frac {T+b}{n+a},\ \frac {\sigma ^{2}}{n+a}\right ),\] and under squared error loss the Bayes estimator is the posterior mean \[\widetilde {\theta } = \frac {\sum _{i=1}^{n}X_i + b}{n+a} .\]
As in the Poisson problem this is a weighted average of \(\overline {X}\) and the prior mean \(b/a\), with weights \(n\) and \(a\); the prior is worth \(a\) observations, and the posterior variance \(\sigma ^{2}/(n+a)\) says the same thing.
Problem 3.4.3. Let \(X_1, \, X_2, \, \cdots \, , \, X_n\) be a random sample from the \(POI(\theta )\) distribution. Assuming the prior distribution of \(\theta \) is \(\theta \thicksim BETA(1/2,1)\), find the Bayes estimator of \(\theta \) using the squared error loss function.
Show solution
Solution. The BETA\((1/2,1)\) prior is \(\pi (\theta )\propto \theta ^{-1/2}\) on \((0,1)\), so with \(T=\sum _i X_i\), \[\pi \left (\theta \mid x\right ) \propto \theta ^{\,T-1/2}e^{-n\theta }, \qquad 0<\theta <1 .\] This is a gamma kernel truncated to the unit interval, not a gamma density, so the normalising constant is an incomplete gamma function. Using \[\int _0^{1}\theta ^{\,s-1}e^{-n\theta }\,d\theta = n^{-s}\,\gamma (s,n),\] where \(\gamma \) is the lower incomplete gamma function, the posterior mean is \[\widetilde {\theta } = \frac {\int _0^{1}\theta ^{\,T+1/2}e^{-n\theta }d\theta }{\int _0^{1}\theta ^{\,T-1/2}e^{-n\theta }d\theta } = \frac {n^{-(T+3/2)}\gamma \left (T+\tfrac 32,\,n\right )}{n^{-(T+1/2)}\gamma \left (T+\tfrac 12,\,n\right )} = \frac {\gamma \left (T+\tfrac 32,\,n\right )}{n\,\gamma \left (T+\tfrac 12,\,n\right )} .\] This is the Bayes estimator under squared error loss. It has no expression in elementary functions and is evaluated numerically.
Remark. The prior deserves comment. A BETA\((1/2,1)\) distribution lives on \((0,1)\), so it asserts with certainty that the Poisson rate is below one. That is a strong claim, and it is what destroys conjugacy — the gamma prior of the previous problem is supported on all of \((0,\infty )\) and returns a gamma posterior. If the restriction is genuinely believed the answer above is correct; if it was chosen for convenience it is the wrong prior, and it is worth noticing that the inconvenience of the answer is the model telling us so.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.