3.3 Conditional Probability Functions and Independence of Random Variables

Question. If \(\operatorname {Cov}(X,Y)=0\) does it mean that \(X\) and \(Y\) are independent?

Results of this type complicate the interpretation of \(\rho \) but essentially it is a measure of the degree of linear dependence between \(X\) and \(Y\).

3.3.1 Special Joint Distribution Function (Bivariate Normal)

\(X\) and \(Y\) are said to have the bivariate normal distribution if their joint probability density function is given by

\[f_{X,Y}(x,y)=\frac {1}{\displaystyle {2\pi \sigma _X\sigma _Y\sqrt {1-\rho ^2}}}\, \displaystyle {\exp \bigg \{-\frac {1}{2(1-\rho ^2)}\bigg (\frac {(x-\mu _X)^2}{\sigma ^2_X}-\frac {2\rho (x-\mu _X)(y-\mu _Y)}{\sigma _X\cdot \sigma _Y}+\frac {(y-\mu _Y)^2}{\sigma ^2_Y}\bigg )\bigg \}}\] for \(\,(x,y)\in \mathbb {R}^2, \quad -\infty < x <\infty , \quad -\infty < y < \infty \), and \(\rho = \) correlation coefficient of \(X\) and \(Y\).

\[\mu _X=E\big (X\big ),\hspace {0.5cm}\sigma ^2_X=var\big (X\big )\] \[\mu _Y=E\big (Y\big ),\hspace {0.5cm} \sigma ^2_Y=var\big (Y\big )\]

If \(\rho =0\) \begin {align*} f_{X,Y}(x,y) & = \frac {1}{2\pi \sigma _X\sigma _Y}\exp \bigg \{\displaystyle {-\frac {1}{2}\bigg (\frac {(x-\mu _X)^2}{\sigma ^2_X}+\frac {(y-\mu _Y)^2}{\sigma ^2_Y}\bigg )}\bigg \}\\\\ & = \frac {1}{2\pi \sigma _X\sigma _Y}\exp \bigg \{-\frac {1}{2}\frac {(x-\mu _X)^2}{\sigma ^2_X}\bigg \}\cdot \exp \bigg \{\displaystyle {-\frac {1}{2}\frac {(y-\mu _Y)^2}{\sigma ^2_Y}}\bigg \}\\\\ & = \frac {1}{\sqrt {2\pi \sigma ^2_X}}\exp \bigg \{\displaystyle {-\frac {1}{2}\frac {(x-\mu _X)^2}{\sigma ^2_X}}\bigg \}\cdot \frac {1}{\sqrt {2\pi \sigma ^2_Y}}\exp \bigg \{\displaystyle {-\frac {1}{2}\frac {(y-\mu _Y)^2}{\sigma ^2_Y}}\bigg \}\\\\ & =f_X(x)\cdot f_Y(y) \end {align*}

where \(X\thicksim N\big (\mu _X,\sigma ^2_X\big ),\hspace {0.5cm} Y\thicksim \big (\mu _Y,\sigma ^2_Y\big )\)

\begin {align*} f_X(x) & = \int ^{\infty }_{-\infty }f_{X,Y}(x,y)dy,\hspace {1cm} \text {Let}\hspace {0.4cm} z=\frac {x-\mu _X}{\sigma _X}\\\\ & = \int ^{\infty }_{-\infty }\frac {1}{2\pi \sigma _X\sigma _Y\sqrt {1-\rho ^2}}\exp \bigg \{\displaystyle {-\frac {1}{2(1-\rho ^2)}\bigg (z^2-2\rho z\cdot \frac {(y-\mu _Y}{\sigma _Y}+\frac {(y-\mu _Y)^2}{\sigma ^2_Y}\bigg )}\bigg \}dy \end {align*}

Let \(v=\frac {y-\mu _Y}{\sigma _Y}\hspace {0.5cm} \implies \hspace {0.5cm} dv=\frac {dy}{\sigma _Y}\)  

\begin {align*} & = \frac {1}{2\pi \sigma _X}\exp \bigg \{\displaystyle {-\frac {2^2}{2(1-\rho ^2)}}\bigg \}\int ^{\infty }_{-\infty }\frac {1}{\sigma _Y\sqrt {1-\rho ^2}}\exp \bigg \{\displaystyle {-\frac {1}{2}(v^2-2\rho zv)}\bigg \}dv\\\\ & = \frac {1}{2\pi \sigma _X}\int ^{\infty }_{-\infty }\frac {1}{\sqrt {1-\rho ^2}}\exp \bigg \{\displaystyle {-\frac {1}{2(1-\rho ^2)(v^2-2\rho zv+(\rho z)^2-(\rho z)^2}}\bigg \}dv\\\\ &=\frac {\exp \bigg \{\displaystyle {-\frac {z^2}{2(1-\rho ^2)}}\bigg \}\cdot \exp \bigg \{\displaystyle {-\frac {1(\rho ^2z^2)}{2(1-\rho ^2)}}\bigg \}}{2\pi \sigma _X}\int ^{\infty }_{-\infty }\frac {1}{\sqrt {1-\rho ^2}}\exp \bigg \{\displaystyle {-\frac {1}{2(1-\rho ^2)}(v-\rho z)^2}\bigg \}dv\\\\ & =\frac {\exp \bigg \{\displaystyle {-\frac {z^2}{2}}\bigg \}}{\sqrt {2\pi \sigma ^2_X}}\int ^{\infty }_{-\infty }\frac {1}{\sqrt {2\pi (1-\rho ^2)}}\exp \bigg \{\displaystyle {-\frac {1}{2(1-\rho ^2)}(v-\rho z)^2}\bigg \}dv\\ \end {align*}

Let \(w=\frac {v-\rho z}{\sqrt {1-\rho ^2}}\implies dw=\frac {dv}{\sqrt {1-\rho ^2}}\implies \sqrt {1-\rho ^2}dw=dv\)

\begin {align*} & = \frac {\exp \bigg \{\displaystyle {-\frac {z^2}{2}}\bigg \}}{\sqrt {2\pi \sigma ^2_X}}\int ^{\infty }_{-\infty }\frac {1}{\sqrt {2\pi }\cdot \sqrt {1-\rho ^2}}\hspace {0.1cm}e^{\displaystyle {-\frac {1}{2}w^2}}\cdot \sqrt {1-\rho ^2}\hspace {0.1cm}dw\\\\ & = \frac {e^{\displaystyle {-\frac {z^2}{2}}}}{\sqrt {2\pi \sigma ^2_X}}\int ^{\infty }_{-\infty }\frac {1}{\sqrt {2\pi }}\hspace {0.1cm}e^{\displaystyle {-\frac {1}{2}w^2}}\hspace {0.1cm}dw\hspace {1cm}\longrightarrow \hspace {0.5cm}\text {the inside is the pdf of N(0,1)}\\\\ & = \frac {e^{\displaystyle {-\frac {z^2}{2}}}}{\sqrt {2\pi \sigma ^2_X}}=\frac {e^{\displaystyle {-\frac {1}{2}\bigg (\frac {x-\mu _X}{\sigma _X}\bigg )}}}{\sqrt {2\pi \sigma ^2_X}}\thicksim N\big (\mu _X,\sigma ^2_X\big ) \end {align*}

It can also be shown that the condition distribution of \(X\) given \(Y=y^*\) is \[N\bigg (\mu _X+\rho \frac {\sigma _X}{\sigma _Y}(y^*-\mu _y),\sigma ^2_X(1-\rho ^2)\bigg )\] , distribution of \(Y\) given \(X=x^*\) is \[N\bigg (\mu _Y+\rho \frac {\sigma _Y}{\sigma _X}(x^*-\mu _X),\sigma ^2_Y(1-\rho ^2)\bigg ).\]

To determine the conditional density of \(X\) given that \(Y = y\), we will continually collect all factors that don not depend on \(x\) and represent them by the constants \(C_i\). \begin {align*} f_{X/Y}(x\vert y) & = \frac {f_{X,Y}(x,y)}{f_Y(y)}\\ & = C1\, f_{X,Y}(x,y)\\ & = C_2 \exp \left \{-\frac {1}{2(1 - \rho ^2)}\left [\left (\frac {x - \mu _X}{\sigma _X}\right )^2 - 2\rho \frac {x(y - \mu _Y)}{\rho _X\rho _Y}\right ]\right \}\\ & = C_3\exp \left \{-\frac {1}{2\sigma ^2_X(1 - \rho ^2)}\left [x^2 - 2x\left (\mu _X + \rho \frac {\sigma _X}{\sigma _Y}(y - \mu _Y\right )\right ]\right \}\\ & = C_4\exp \left \{-\frac {1}{2\sigma ^2_X(1 - \rho ^2)}\left [x - \left (\mu _X + \rho \frac {\sigma _X}{\sigma _Y}(y - \mu _Y\right )\right ]^2\right \} \end {align*}

From the last equation we can conclude that, given \(Y = y\), the random variable \(X\) is normally distributed with mean \(\mu _X + \rho \frac {\sigma _X}{\sigma _Y}(y - \mu _Y)\) and the variance \(\sigma ^2_X(1 - \rho ^2)\).

3.3.2 Practice problems

Show solution

Solution. Here the success probability of a binomial is itself random – a mixture. The tools are the tower property and the conditional variance formula: \[E(Y) = E\big [E(Y\mid X)\big ], \qquad \operatorname {Var}(Y) = E\big [\operatorname {Var}(Y\mid X)\big ] + \operatorname {Var}\big [E(Y\mid X)\big ].\] Given \(X=x\), \(Y\) is binomial, so \(E(Y\mid X) = nX\) and \(\operatorname {Var}(Y\mid X) = nX(1-X)\). For the beta distribution, \[E(X) = \frac {a}{a+b}, \qquad \operatorname {Var}(X) = \frac {ab}{(a+b)^{2}(a+b+1)}.\]

(a). \[E(Y) = E(nX) = nE(X) = \frac {na}{a+b}.\]

(b). Take the two pieces in turn. Since \(E(X^{2}) = \operatorname {Var}(X)+\left [E(X)\right ]^{2}\), \[E\big [\operatorname {Var}(Y\mid X)\big ] = n\left [E(X)-E(X^{2})\right ] = n\left [E(X)-E(X)^{2}-\operatorname {Var}(X)\right ],\] \[\operatorname {Var}\big [E(Y\mid X)\big ] = n^{2}\operatorname {Var}(X).\] Adding them and writing \(s = a+b\) for brevity, \begin {align*} \operatorname {Var}(Y) &= n\left [\frac {a}{s}-\frac {a^{2}}{s^{2}}-\frac {ab}{s^{2}(s+1)}\right ] + n^{2}\frac {ab}{s^{2}(s+1)}\\ &= \frac {nab\,(s+n)}{s^{2}(s+1)} = \frac {nab\,(a+b+n)}{(a+b)^{2}(a+b+1)}. \end {align*}

This is the beta–binomial distribution. Compare it with an ordinary binomial having the same mean, whose variance would be \(np(1-p) = \frac {nab}{(a+b)^{2}}\): the mixture carries an extra factor of \(\frac {a+b+n}{a+b+1}\), which exceeds 1 whenever \(n>1\). Letting the success probability vary makes the count more variable than a plain binomial – this is overdispersion, and it is why the beta–binomial is reached for when count data are more scattered than a binomial can explain.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.